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This module discusses the evaluation of the long-term ^ 
effectiveness of curricula in preparing students for respon- 
sible performance in the world of work. Vocational , education 
evaluation should begin in the curriculum planning stage and 
continue during the developmental, early implementation, and 
post-implementation sta^^. Once a curriculum is implemented 
and found to be successful using in-school evaluation measures, 
follow-up of graduates working i^ their occupational fields is 
advisable. Information obtained from such follow-ups can be 
Used to improve the curriculum. 

1 

The relatively recent call for accountability in educatiort 
has necessitated evaluation of teachersf studisnts, administra- 
tors, institution? , and curriculum. In order to participate in 
this process, the vocational education curriculum specialist 
must be familiar with the general prino^lples of evaluation and 
must be able to conduct or supetvise curriculum evaluations. 
He or she most ^aisp know how to use evaluation data to improve 
the curriculum. i > 

• J _ . . 

Because folJb6w-up evaluations idehtify needed curriculum 
revisions, this module is related to the modules in this series 
on needs assessment, curriculum, design, development, and change. 



Overview * ' 

The first goal of the module 'Ts* to summa*ri^e th^ major 
activities included in planning for a follow-up evaluation^ x ^ 
Emphasis is' placed on clarifying the purpose of this part <^ a 
curriculum evaluation, establishing the scope and objectives of 
the study, and preparing a woi'k plan. - ^ 

* Goal 2 focuses on the actual conduct of a follow-up study. 
Suggestions ar^ given for sampling, developing the data collec 
tion instrument, conducting the data collection, and processing 
and analyzing data. \ 



Goal 3 provides guidelines for interpreti^ and communicat- 
ing evaluation results* The team approach, to drawing conclusions 
and making recommendations is introduced, and a. comprehensive 
evaluation ref)Oxt <j€rt:line is presented. Commutncation strategies 
that will maximize implementation of recommendations are also 
described. 



Instructions toUhe Learner 

— ' ^ ' , . ^ 

The Self-Check it^ras and possible responses to them are 
found in the appendices. Jheae questions have two purposes. 
First, before you begin work* on the module, you may use them to 
check quidkly' v/hether you have* already learned the^ information 
in previous classesv.or readings. In some* instances, .with the 
consent of your instructor', you inight decide^ to skip a whole 
module or parts of one. The second .purpose of the Self-Check 
is to help you review the content of modules you have studied 
in"'order to assess whet;her you have achieved the module's goals 
and objectives. # 

You can alsd use the list of goals and objectives that 
follows to determine vhether the modjiJ?^\pontent is new to .you 
and requires in-depth study, or whether %he module can serve a« 
a brief review before you^ continue to the next module. 



Goals and Objectives 



Goal I s Summarize the major activities included in the initial 
planning for 'a follow-up evaluation. 

Objective 1« 1 ; List the purpose for conducting a follow- 
^ up study as part of a curriculum evaluation. 

Objective 1.2 : Describe the proce^ss of establishing the 
objectives of a particular follow-up evaluation. 

^ Objective 1.3 : List and describe the coritents of the work 

>plan for a follow-up evaluation. 



Goal 2 : Summarize the major ac'tivities included in the conduct 
of a follow-up evaluation. 

Objective 2.1 : List 'guidelines for selecting or con- 
structing a data collection instrument. 

Objective 2. I x Describe the steps in conducting a follow- 
-up evaluation, including (a) selecting the sample; (b) ori- • 
enting data collection staff; (c) locating and (Orienting \ 
potentia;! part'icipants ; (d) collecting data; and (e) fol- 7 * 
^ lowing up noni^espiondents. 

Objective 2.3 : Describe the' process of processing, anal-' * ^ _ 
yzing, and p^resenfing data. ^ / 



Goal 3 : State basic guidelines' for interpreting the results of 
a curri\:ulum evaluation and for communicating these effectively. 

Objective 3.1 : . Describe the steps involved in drawing 
conclusions arfd making recommendations from curriculum 
evaluation results.' . ' ^ 



Objective 3.2 : Uescribe. the majar content elements that ^ 
should be included in a report or presentation of 'curricu- 
lum evaluation re'sults. 



Objective 3.3 ; St^ate basic guidelines for effective com- 
munication that should be followed in preparing a report 
or presentation of, curriculum evaluation results. 



\ * 
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Resources » , , / 

In order to complete' the learning activities in this ^ 
module, you will need informatiqn contained in the following 
publication: " , . • 

i 

Wentling, T. L. Evaluating occupational education and 

training programs . Boston, MAr Allyn and Bacon, 1980. 



'GOAL 1 "Summarize tfie nJ^jor activities included in the initial 
planning for a follow-up evaluation.*' 



Planning Foll ov- Up E valuations 

4 

In addition to conducting an evaluation of the shbrt-term 
outcomes of a vocational education curriculum, 'it is important 
to evarluate the long-term effects of curriculum oa the knowledge 
and performance T^f graduates. To do this, curriculum special- 
ists conduct^ follow-up studies of former students and their 
employers . Even if the use of a particular, curriculum has pro- 
duced significant gains for participants based on pre- and 
posttest scores, broader questions remain for the curriculum 
specialist: What are the long-term effects of the curriculum? 
Are the objectives and c^tent^of the curriculum enabling stu- 
dents to meet the, demands of tfte /working world? Does a stu- 
dent's mastery of course objectives indicate alikelihood of 
succes^s^l job placement and performance? How can the curricu" 
lum be improved? » j * 

Information collected from former eftudents and their 
employers through follow-up studies can help answer these ques- 
tions. Careful planning of follow-up studies can help assure 
that they provide adequate information. - ' ' 



Purposes of Follow-Up Studies 

W^ntling (1980) lists 'nine purposes for conducting a ^ 
follow-up study of vocational pi/ogram graduates. They are:* 

determining career patterns, 

determining job availability, ^ 

determining 'graduate mobility, 

determining program adequacy for purposes of job entry-j 



determining program adequacy for purposes of entry 
into advanced training, \ 



determining] the adequacy of ancillary serv^es sych as 
H guidance or placement, * *' 

.emphasizing career objectives, and 

• providing information for required reports. 

^ * • 

Wentling also lists seven purposes for conducting an 
employer follow-up. These include: ^ ^ 

• assessing the performance of former studen^ts, 

« 

^ • determining how specific program graduates compare 
with other graduates, 

• obtaining recommeridations for improving the program, 
^ • determining recruitment practices, 

• assessing competency lists, 

• estimating worker supply and demand in particular 
occupations, and 

t * 

• enhancing public relations. 

Other purposes can be added "^to Wentling 's lists. FolTow- 
ups could be conducted to improve vocational instruction, or to 
.demonstrate the value of vocational programs to groups provid- 
^ng financial support. 



The information and suggestions provided in this module 
are appropriate to any one of .these purposes, but the focus of 
this module will fee restricted to those purposes of follow-up 
studies most central to the duties. of a curriculum specialist: 
evaluation to improve the curricuj.um or judge its worth . 

Establishing the ObjectiV&s of a Particular Follow-Up 

The timing of a follow-up stiMy may be influenced by a num- 
ber of factoV:i5::rthe need to write a program rationale in order 
to receive federal or state aid; the raising of various types . 
of questions about a curriculum by the Board of Education, 
teaching staff, or.' community members; or the curriculum special- 
ist's personal determination that information is needed on a 
certain curriculum's long-terra impact. For example, a new 
energy curriculum may have been implemented three years ago 4nd 
the curriculum specialist may believe that it is time to conduct 
a follow-up study of its impact on graduates' job performance.. 
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Because evaluations are often conducted to meet the needs 
of several individuals or groiips, it -jis possible that the cur- 
riculum specialist may participate in a follow-up designee! to 
meet more than one purpose. However, the general focus of this 
module will be on vocational education curriculum evaluation . 

The process of establishing the objectives of a follow-up 
evaluation consists of three steps; 

• Convening a planning meeting and selecting an evalua- 
tion director 

• Defining the scope of the study 

• .9?riting the general goal of the follow-up and listing 
primary and subordinate objectives ^ 

. The initial step in conceptualizing a follow-up study is 
to convene a small planning meeting of selected school adminis- 
trators, school board or advisory committee members, guidance 
personnel, teacher-s, and present vocational students. The pur- 
pose of the meeting is to choose a leader fqr the follov"up 
effort, define the scope of the study, and develop a list of 
study objectives, based on a tonsensus of the curriculum 
evaluation needs^ftand concerns of1:he group. 

The importance of a strong evaluation director--presumably 
the curriculum specialist — cannot be overemphasized. It is not 
assumed that this individual will possess technical expertise^ 
in all areas of evaluation since other individuals within the 
•school system may be tapped fqr their specific skills. Jt is 
essential, however, that one individual, whose mind is firmly 
set on the goals of the evaluation, coordinate all staff, 
activities, see that tasks are completed correctly and on 
schedule, and be committed t\) communicating evaluation results 
effectively and widely in order to maximize implementation of 
changes, resulting from the evaluation's findings. Developing a^ 
spirit'of communication and cooperation early among staff is' 
also important, ^ince it invariably results in a broader, more 
relevant research design, more efficient operation, and greater 
receptivity 'to and implementation of findings. ' 

In regard to peJrsons to be involved in the evaluation 
planning effort, it is often valuable to obtain the support of 
a pre-existing citizens' advisory committee ,or a panel of 
several prominent community members constituted specifically . 
for this study. Since thes6 individuals will be involved in ^ 
the "world of work" and may be supervisors of large numbers of ^ 
employees themselves*, they will provide helpful input in the 
•conceptualization of the study, the development and pilot test- 
ing of the data collection instrument, and the contacting of 
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employers- ,They will also be strong '^dvocatea of cuirriculum 
revisions based on study findings that promise to improve stu- 
dents' jferformance in the work setting. - 



The plarming group must determine the scope of the follow- 
up .study. "Tnek group must decide the specific vocational curri- 
culum to bel^luatgd (e.g., the automotive mechanics 'curriculum 
implemented in 1975) and which groups of individuals will be 
asked to provide information to the follow-up. In a student 
follow-up, the findings will be enriched by efforts to include 
farmer stuTfent'S vho successfully completed the vgcatiilnal edu- 
cation curriculum of interest) *Mropouts,^' and persons in other< 
secondary curricula Vho took occasional courses within the tar- 
geted area. Former s.tudent« who currently are working, who are 
enrolled in advanced educati6nal institutions, and who are 
unemployed should all be surveyed, though some groups may be 
more difficult to locate or more reluctant to respond. It may 
also be helpful to survey persqn^ of different ages and at dif- 
ferent stages of job development who have completed the same 
curriculum or the twq cv^rricula tCK be compared (e.g., the old 
and the new automotive mechanic^curricula) . 



-The planning group will Vlso need tct decide whether the ' 
follow-up will assess Students* perceptipns of the quality and 
appropriateness of their training or employers* satisfaction 
with graduates* job' performance. Often information should be 
sought from both students and empToyers. , 

Graaso (1979) refers- to a'^Varticular vocational education 
. follow-up, study in which former vocational students rated their 
high school curri'cula as highly successful in preparing them 
for occupations, The ,same study, however; revealed that other 
measure*^ of thfe program's effeci^iveness , "including ratings by 
employers of students* on-^'th*|- job .performance, were less than 
satisfact^ory. ' This suggests 'that graduates* ratings of their 
programs may not be the bes^f method of evaluating vocational 
graduates* achievement of program objectives iln terms of satis- 
factory job performance, The ^employer fdllow-up study is a 
more objective measure of , this and, as such, is a useful c;om- 
plenTent to the student follow-up. V 

If the sciope of the curriculum evaluation is to include an 
employer follow-up, the exact employer g^oup to be surveyed 
shoutdialso be established. Generally, employers will be those 
currently-supervising grsfdu^tes of the particular occupational 
program of iriterest (e.g., emp-loyers of program graduates who 
studied the 1975 automotive mechanics curriculum), 

A general goal for t)ie follow-up should be written to sum- 
marize the discussion of the scope of the stMdy. From the gen- 
eral goal of the stuMy, primary and subcxrdinate objectives 
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should *be written. Foi example, a* student follow-up e>aluatio(r0 
focused on curriculum validation^ and revisicyn* might have the ^ 
goal and objectives shown dn Fdfgure 1. An employer follow-uj) 
evaluation might have the, goal and objectives shown in Figure 2. 



Sample Set of Objaccives for a Student Follov-Up 
Study Aimed at Currlculua Evaluation 



— ' : r 

Goal ; To judge the overall value of the secretarial curriculua 
in preparing students for work, and to i^iptove the cur- 
riculum as needed. 

Primary Objective #1 : To determine whether students are 
seeking -^nd obtaining clerical and secretarial jobs. 

Primary OSjective #2 : To determine whether students are 
advancing in thes6 Jobs. 
* 

Prlnary Oblectlve ^3 : To determine whether students feel 
that they possess adequate on-the-job skills to perform 
^ satisfactorily as clerks, typists, secretaries, and steno- 
graphers, o 

Subordinate Objective ^I : To determine whether 
' students feel that they possess the technical skills 
needed In clerical/secretarial occupations (typing » 
shorthand, etc.)*' 



/ 



Subordinate Objective #2 : To determine whether 
students feel that they possess the general academic 
skills needed In, clerical/secretarial occupations. 

Subordinate Objective #3 : To determine whether 
students feel that they possess the interpersonal 
skills needed in clerical/secretarial occupations. 
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Figure 2 ^ , 0, 

Sanple Set of Objectives for an Eoplpyer FollowUp 
Study Aimed at Curriculum Evaluation 



'^iT: To Ji^g^ the overair value oi the food service curriculum In ^ ' ' 
preparing students for work, land to Improve the currlctilum as 
needed* / 

Primary Objective ^I : To ascertain employers' expectations 
for entry-level food service workers, ^ ^ 

Subordinate Objective ^I : To obtain a description of 
the job (containing a list of taslks). 

Subordinate Objective ^2 : To obtain a list of personal 
qualities and job skills considered most important in 
a person entering the job. 

Primary Objective ^2 : To determine whether graduates possess 
\ adeqt4ate on-the-job skills to perform satisfactorily as short- 

order cooks, food preparation workers, waiters and waitresses, 
etc. ^ 

Subordinate Objective #1 : To dec^ermine whether students 
possess the technical skills needed in these occupations 
(buying provisions, oeal planning, food preparation, 
etV:.). ^ 

Subordinate Objective ^2 : To 4etermine whether students 
possess the general academic knowledge needed In these 
occupations (reading, computation, etc.)* 

Subordinate Objective 03 : To determine whether students 
possess the interpersonal skills naeded in these occupa- 
t ions . 

J 

Primary Objective 02 i To ascertain employers' satisfaction with 
graduates of this particular coimserclal home economics curriculum. 

Subotdinate Objective s^l : To determine how graduates of 
this curriculum compare to gradxiates of similar cAirricula 
and to persons with no prior vocational training. 

Subordinate Objective #2 : To obtain specific suggestions 
for changed In this vocational curriculum. 




The preparation of specific objectives for the* follow-up 
is a crucial task in planning for the conduct* of the evaluation. 
The discipline necessary to state detailed subordinate objec- 
tives will be rewarded when the task of selecting or developing 
data collection instruments is initiated. It is necessary to 
have a coraprehensdve set of subordinate objectives that reflect 
the curriculum revisjj^n decisions for which the follow-up must 
provide data^ In this way, the data collection instrument can . 
be designed to collect the necessary infoi^mation. Primary and 
subordinate objectives should be approved by the planning 
groups that helped determine the scope of the evaluation. _ 




Preparing the Work Plan i 

Th^_. importance of a work plan that outlines the goals and 
objectives of the study, the tasks and products involved, ^nd 
the personnel and resources required andxthat links these to a 
schedule of completion dates cannot be overemphasized. 

serves to guide the daily aativities of all staff 
course of the evaluation. It a\ffo informs interested 
is of intended activities, and focuses the attention of 
all to the ultimate goals and products of the effort. 

\ • / 
The work plan should include specific information for each 
task necessary to complete the follow-up evaluation. In^ gen- 
eral these tasks include: ^ 

4 " - 

1. Defining the goals and objectives of the evaluation " 

2. Preparing a work plan and obtaining approval to pro- 
ceed with the study m ^ 

3. Selecting the method of data collection 

4. Selecting<?or constructing and pilot testing the data 
collection instriiment(s) » 

5. Selecting^he sample 

6. Orienting the data' collection staff * 
- 7. Locating and orienting potential participaihss 

8. Collecting the data 

9 . Contac t ing, nonrespondent s - - 

10. Processing, analyzing, and presenting data 

-21- ♦ 
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^11. IXrawing conclusions and making recommendations 

12. 'Developing a strategy for communicating evaluation 
findings ' * / 

13^ Preparing, ari evaluation report or presentation. 

In developii)g an overall work plan for the study, the 
following elements . should *be included for each task: 

1. Goals and primary objectives . Linking this informa" 
tion to the more practical elements of the work plan 
helps ensure that the ultimate uses of the study find 
ings are k3pt in mind as the individual tasks of the 
study are carried out. ^ 

2. A list of tasks . It is helpful to arrange project 
tasks in flowchart form for easy reference. Tasks 
should als'o be defined in greater detail elsewhere, 
with subtasks enumerated as appropriate. In the data 
collection phase, for example, several waves of nonre 
spondent follow-up are needed; and during data pro- 
cessing, several steps are also recjuired. 

3. Products cff each task . Products may include reports; 
data collection instruments; letters, ^mes, and 
addresses of sample participants; and publicity items 

4. Completion dates for taaks . Dates should be clearly 
defined to facilitate coordination of staff schedules 
to permit avoidance of holidays and other busy times 
of the year for data collection, if possible; and to 
§llow efficient cooperation with off-site services^ 
such as computer facilities. 

5. Individual (s) responsible . These may include Xby job 
title) administrators, 'Board of Education members, 
teachers, guidance personnel, support staff, and cur" 
rent or former students; and (by function) Study 
director, persons to design instruments, data collec- 
tors/pcocessors/analysts / persons to locate sample 
participants, individuals responsible for project pub 

^ licity, clerical and secretar ftfl«*^|tf f , and computer 
and printing staff. 

\ - . _ 

6. Resources needed . These may include staff and consul 
t^nt salaries, supplies and operating expenses, work 
and storage facilities, and computer time. 




A work plan sununary can be prepared by listing these ele- 
meftts as column headings across the top of a sheet of paper and 
then filling in the columns with the appropriate information. 
The work plan summary, along with a brief rationale paragraph 
stating why and how the evaluation was conceived, by whom, and 
the^>curriculum and population to be studied^ should be sub- 
mitted to the appropriate administrative officer or board in 
order to secure project approval and administrative and finan- 
cial support. 

Selecting the Method of Data Collection 

An important decision in preparing the work plan is select- 
ing an appropriate method of data collection. The methods that 
may be used include: personal interview, telephone interview, 
and mail questionnaire* 

Personal interview . The advantages of the personal inter- 
view are that: it results in a high rate of response; it is 
very effective in developing public relations; and it allows 
for collation of more detailed information from respondents 
through interviewer probes. Disadvantages of this method 
include increased expenditures in terms of staff time, staff 
salaries, and travel costs. Additional time is required for 
supervisory personnel in locating and training interviewers; 
for interviewers in locating and talking with participants; and 
perhaps for data processors and analysts. Data analysis is 
particularly difficult when staff must evaluate and classify 
explanatory responses. Travel costs for face-to-face meetings 
with persons living out of town or out of state are also sub- 
stantial* 



Telephone interview * The telephone interview has many of 
the advantages of the personal int e rview. The rapport— estab- 
lished through personal pjjone contact results in high partici- 
pant cooperation and ability to obtain unambiguoqs and compre- 
hensive responses to items. The prime disadvantage here, too, 
i^s the high cost of staff training, long-distance phone calls, 
^d data processing. These two methods kre probably most valu 
able when smaller samples are being used, when the development 
of good public relations is of high priority", or when low moti 
vation levels of the target population are suspected* 



Mail questionnaire ^ The mail questionnaire is, on the 
other hand, a lower-cost method of data collection that is 
effective for contacting persons in a wide range of geographi- 
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cal locations. Because mail questionnaires are more impersonal 
in nature^ certain ih^formation can actually be obtained more 
readily through this method, such as information about partici- 
pant income. However, participant responses to items cannot be 
readily clarified or expanded, and the mail follow-^up^ generally 
has a lower rate of participant cooperation. Even with its 
disadvantages, the mail questionnaire is generally the pre- 
ferred method of data collection in follow-up studies. Use of 
t^his method assumes that a more intensive strategy for securing 
respondent participation will be included in the work plan.' 

The pros and cons of each of these data collection methods 
should be weighed against the purposes and information needs of 
the study, the size of the sample, and the resources available. 
The nature of the data collection instrument will vary accord- 
ing to the method selected. 
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Individual Study Activities 

1. ^ Read pages 140-146 and pages 182-189 in Wentling, T. 

Evaluating occupational education and training programs^ , 
Boston, MA: Allyn and Bacon, 1980. 

2. Assume you believe that a follow-up evaluation should be 
conducts^d. Yoxi thitik the follow-up should bS ^airaed at 
improving the physic^l^therapist assistant (PTA) training 
curciculura at. the local junior college where you are 
employed as a curriculum specialist. You want to convene 
a mee^ting to get planning for the study off the ground. 

a. List the types of individuals you will invite to the 
meeti-ng. Briefly state why you will invite them. 

b. List the types of significant decisions you hoi^e will 
result from the meeting. 

3. Prepare a flow diagram illustrating how the tasks that 
should be included in the work plan of a follow-up curri- 
culum evaluation could be scheduled and coordinated. Put, 
each task in a box and draw arrows betweeri boxes to indi- 
cate relationship betwe^en tasks. Let the time dimension 
run from left to right on your page to illustrate that a 
task is completed before those shown to the right of it 
and after those shown to its left, , * 

4. The following are statements describing methods of data 
collection used in follow^H^o studies. For each statement, 
circle T if the statement i^s^rue and.F if it is false. 

T F !♦ An advantage of the mailed questionnaire i^ 
that you can contact a large number of per- 
sons (e.g., 100) with the l^ast expenditure 
of time. ) ^ « 

\ 

T F 2. The personal visit permits g&thering of 

informat^n in more depth than the mailed 
questionnaire. * 

T F ,3. The telephone survey provides a means of 

checking and assuring the correct interpre- 
^ tation of qu€iStions. 

i 

T F \4. The mailed questionnaire is likely to ^have 
a higher response rate than the telephone 
survey. 
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F 5. Using the personal visit to conduct employer 
follow-up* allows the evaluator to promote 
^hiring of new program'graduates at the same 
time follow-up information is being col- 
lected. -T) 

6. An effective way tp increase response rate 
is to follow up a mailed questionnaire with 
a telephone survey pf nonrespondents. 

7. The telephone , survey has similar advantages 
to the personal interview buj: is less 

__aostly. 



T F 8. An advantage of the mail questionnaire is 
that much of the data processing can be 
'done by clerical personnel, * thus freeing 
th*fe evaluation director for other activi- 
ties. 



Discussion. Questions 

^ ' ^/ . . 

1. What student or employer follow-up studies are you person- 
ally familiar with? What ^ere the goals of the study and 
how vas it designed? Did the study achieve its goals? 
What actions werfe taken based on the recommendations^re- 
senteii? What problems in Administration of the study were 
encountered , and how were these solved? ^ - 

2. Are you familiar %fith any school districts where voca- 
tional follow-ups have not been conducted in the past five 
ye-ars? Based on the information you have of a particular 

^vocational education curricultim (obtained from informal 
contacts with former students and their employers in the 
community) J what degree of success do you think the curri- 
culum is having , in helping students perform well in their 
later jobs? Is your current information sufficient to * 
draw reliable conclusions? Discuss whether a systematic 
follow-u| study would be useful and feasible at this time. 

Group Activity • ^ * ^ 

*, » 

In groups of fout or fi%, choose a particular vocational edu- 
cation curriculum and then plan a student (mail) fojlow-up 
study. The emphasis of this activity is on research design and 
planning of tasks and procedure^i; No actual data collectipn 
will be conducted in this exercise. 
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Define the group of former students you want to survey. Develo 
primary and subordinate objectives for the study. Determine 
the method of data collection you will use. 

Then complete the wqrk plan form, provided, enumerating tasks, 
products, time schedules, and responsible staff. In the space 
lab'eled '"Individual(s ) Responsible," give the functional title 
of schpol district personnel who might perform each p'Ssk. 
Develog^a realistic, time schedule £or all tasks based on the 
amoTOt of ti^ie you th^nk it would take school personnel to com*- 
plete- each task in a full-blown version o^ this study!.* 

Your work group should complete these products: 

t 

• • A paragraph describing the purpose of your study and 

the proposed study participants ' 

• list of primary and subordinate objectives 

• A project work plan summary (time/task/talent chart^ 
^(form provided) ^ - " 



WORK PLAN 



Tasks and Sub tasks 
(dumber. gach major task) 



Products of Tasks. 




Aoount of 'Time Rdaulred 
for Each TasK 



0 2d 



„ Individual (s) Responsible 
for Each Task 



I ERIC 



GOAL 2: Sxxcmatize the major activities included in the conduct 
of a follow-up evaluation. 

— - — ^v—. 

r 



Conducting a Follov-Up Evaluation 



The preceding goal (discussed the planning of a follow-up 
study of students and their employers. The focus of this goal 
is on actually conducting the follow-^ study. Tasks required 
to conduct** the study include: select Ag or Constructing a data 
collection instrument; selecting the sample of study respon- 
dents; orienting data collection staff; locating and orienting 
potential participants; collecting the data; following up non- 
respondents; and processing, analyzing, and presenting the data. 

Selecting or Developing the Data Collection Instrument 

The steps' ^required in developing a data collection ^instru- 
ment include: 

• identifying existing data Collection instruments; 

• selecting the* appropriate existing instrument or 
developing a new one; and ^ 

• pilot testing the instrument. ' • 

Existing data collectic/h instruments useful' in follow-up stu*- 
dies c^n^ be located through any combination of the following 
sources: \ . 

• The director of the State Research Coordinating Unit 
for" Vocational Education (existing in all 50 states)'* 

• Th^ State Department*of Education, Division of Voca- 
tional-Technical, Education 



A school adniinistrator or dSector o'f vocational edu- 
cation in local school systems (your own district, a 
nearby system, or a large Urban system within the 
state) 
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Industrial psychologists in universities and industry 
Literature searches of comput erized data bases (such 



as ERIC, Dissertation Abstracts , Psychological 
, Abstracts , Sociological Abstracts , and the Library of 

the National Center for Research in Vocational Edu- 
cation) ' ^ 

• Source books of data collection in/truraents, such as 
Buros Mental tleasurement Handbook (1978)* and a hand- 
book developed' at the National Center for Re^earcli in 
Vocational Education by Gray, Abrara, McKinney, and 
Billings called Vocational Education Measures: Instru- 
ments to Survey Former Students and Their Employers , 
1978. ' ^ 

^^Irhe curriculum specialist should use criteria such as 
those listed below to evaluate existing instruments for pos- 
sible use. 

• Is the instrument readily available at minimal cost? 

• Is the instrument appropriate -to th6 target population 
of the study (e»g,, is it suit-able to following-, up 
graduates of the auto mechanics program)? 

/ • Is the irtstrument written to meet the same goal(s) and 
objec'^tiyes as those of the propos^ed study, and does it 
have sufficifent coverage on topics of interest?, 

• Is the instrument suitable for the time frame of the 
study' and the method of administration selected (efg»> 
one-year follow-up survey by mail)? 

• Is the item format suitable foo: efficient collection 
♦ of data', and is tKe style clear and concise? 

• Can participants complete the instrument in a reason- 
able amaunt of time? 

• Can data be processed in an easy, cost-effective 
manne|!? 



* This handbook contains primarily knowledge and ijiterest tests, 
' but it also lists a few questionjagires designed to measure 
employees' job satisfaction and performance. 




• Is information available about the instrument s reli- 
^ ability? (Will responses to items be the same if the 

instrument is administered a §,econd time to the same 
group Ql-jceBjondentA?) 

• Is i-nformation available about the instrucaf nt ' s vali- 
dity? (is its content representative of the topic 
area to be measured?) 

An existing instrument should be chosen for use in a par- 
ticular follow-up only if it efficiently collects the ''informa- 
tion necessary to meet the goals and objectives of the study. 
Often, existing instruments will be inadequate to use "as is.'l 
Their general format may be satisfactory, but it is likely that 
only certain items will match the planned study's objectives.' 
If copyright permits, specific items may be borrowed from one 
or i^'^re instriimehts add assembled, along with original items, 
into an irijBtrument that is totally suitable for the follow-up 
study to be performed. 

In constructing the data collection instrument, the pur- 
pose of the evaluation should 'be held clearly in mind> and items 
should be designed specifically to provide information to meet 
one of the stated objectives of the study. Generally, items 
should be grouped by objective (e.g., all items related to a 
former student's general job history should appear together). 
The instrument should be arranged with the^most important 
objectives first, or according to another logical sequence,. 
While keeping the instrument as short as possible, the evalua-* ' 
tors must make sure that sufficient information is obtained so 
that behaviors may be explained as well as described. For 
example, besides obtaining the information that a particular 
individual was unemployed for a year after high school, the 
instrument should contain an item documenting why this occurred *^ 

The instrument should be reviewed by those people who will 
process and analyze the study results before its final content 
And format is set. 

The format of items (dichotomous , multiple-choice, rating, 
ranking, and open-ended items) should be determined from the 
type of information desired. Varying item format or, at times, 
simply varying wording among consecutive items with the same 
format is important for retaining respondent interest and 
attention to the precise meaning of each question. 

AlsO) if the evaluation is part of a multiphase study to 
be conducted over several years, care should be taken to col- 
lect identical or comparable information using the same or 
similar items for each phase of the evaluation. Otherwise, it 
will be impossible to make comparisons between years* 
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Two types of ins,truraent directions are necessary. Overall 
directions should be presented at the beginning of the instru- 
ment. These should state the purpose of the survey and, if 
maileci, the desired return date for the instrument. Individual 

-diiree^OR8-sh ould - bo - plac e d wi t hin . th e, .insrrnmpnfi at tho afi 

points where item format changes, The^e should orient the 
respondent to answering procedures. 

Questionnaires that are att^tive, use high quality^aper , 
•have a spatious layout, ^nd are clearly printed will encourage 
a high response rate. The use of colored i^aper has also been 
shown to increase the response rate. .Items should be readable 
without having to turn the questionnaire sideways. If printing 
is to be sm^ll, the use of elaborate tjpe faces should be 
avoided. Other problems related to instrument appearance can 
be avoided if the evaluator simply imagines his or her own 
reactton to a mes^y, unreadable, and dull questionnaire. 

Other strategies for maximizing response by mail include 
placement*of difficult questions at the end of the instrument 
and avoidance of items requesting information that is already 
available from other sources or that may possibly be considered 
confidential or irrelevant to the purpose of t^study. One 
such "sensitive" question might be an item asking the respon- 
dent's religious affiliation. 

A pilot test of the instrument with a few se'lected indi- 
viduals before administration i:o the target population will 
point up laok of clarity in directions, unclear terminology, 
problems of interpretation, and weaknesses in the format. It 
will also serve as a tryout of procedures for instrument admin- 
istration and data processing. Revisions of the instrument 
should be made ^s* needed after the pilot test. 



In choosing an existing instrument or developing a new 
one, the following guidelines should'be considered: 

'^^0 Keep in mind the overall goal(s) of the evaluation 

• Develop/select items to meet^ specif ic subordinate 
objectives * \ 

• Group related items and arrange gr^s in a logical 
sequence 

• Word items clearly to avoid misinterpretation 

• Keep items and'options concise, and vary their format 
t<;^^^enhance interest 
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• Avoid asking unnecessary, confidential, of "sensitive" 
questions 

• Provide clear. instructions on how to respond to items 



Additional guidelines relating speciticaily to the mail ques*^ 
tiorinaire inclu^de following: 

• . As much as possible, use items in which a limited 

nutaber of options are listed (i.e., dichotomous,* 
multiple-choice, ranking, and rating items> 

• Use open ended litems wisely te.g*, as summary ques- 
tions, to collect any additional data, or in cases in 
which possible answers to a question are not easily 
predicted; 

• Design the instrument to have an orderly layout and 
attractive, appearance 

Selecting the Study Sample 

Depending on tf^^urposes of the study, the evaluator may 
want to survey graduates of a particular vocational program or 
their employers or both groups. Based on the size of the tar- 
get population and the scope and resources of the study, it may 
be appropriate to choose a sample of the population to partici- 
pate in the followup.. If random sampling techniques are used, 
study participants are likely to be truly representative of the 
whole target group. 4 

In random sampling, every member of the pppulation has an 
equal chance of being selected^ Ihe sample is chosen through 
one of several random selection methods — by lottery, by flip- 
ping a coin, by randomly selecting names from an alphabetical 
list, or by randomly assigning each person in the target group 
a number and then picking numbers from a table of random num- 
bers.* Because selection is based on chance, the assumption is 
made that, given a sufficiei^t number of * individuals, persons 
selected will possess the same variety of psychosocial beha- 
viors, academic ability and motivation, and job performance and 
satisfaction as the entire population. Thua^ generalized 
statements made about the whole group based on data collected 
from the sample can be defended* According to Young and Schuh 
(1975), for populations of 600 to 5,000, a sample size of five 
•percent is considered appropriate and safe* For populations of 
less" than 600, at least 30 rafidomly selected respondents should 
*be included in the study. ^ 
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When subgroups within a population affrof interest le.g., 
males and teraales}, the population should be broken into appro- 
priate subgroups, and a random sample, of each subgroup, known 
as a ''stratified random sample," should, be , taken and included 
in the to 1 low-up study. 

Sampling procedures should be documented car'efully so that 
the evaluator can argue convincingly in the evaluation summary 
that the persons surveyed wefi indeed representative of the 
entire target population. // 



Oriehting Data Collection Staff ^ 

Data collection staff will need to be trained to locate 
arid contact study participants. Staff members should learn 
strategies for locating pteopl^nd for keeping records once 
individuals are contacted and (^rolled in the study. 

tf the interview method is used^H-t may be necessary to 
instruct staff members in proper and effective procedures for 
interviewing, including introducing the survey, asking ques- 
tions, probing for clarification and additional information, 
and recording Responses. A standard introduction to the inter- 
view should be developed ahead of time that indicates the sur- 
vey purpose and is aimed at establishing r^lJport between intern- 
viewer and respondent. Interview persprlnel should be able to 
present^^is introduction .smoothly arid also should be entirely 
familiar with the survey ^instrument prior to conducting the 
firs^ interview. 



Locating and Orienting Potential Participants 

Once the names of the individufiUs who wilt participate in 
the study have been listed and the data collection staff has 
been trained, the current location of participants must be 
deteiWlined. ^ - 

Fx>rmer students may have moved several times since high 
school. Their addresses may be obtained through a variet^ of 
sources such as school counselors, parents, post office 
records, and phone directories. 

The orientation of students is a crucial step in securing 
their participation in the study, since many factors relating 
to lack of motivation and lack of comprehension encourage non- 
response. One orientation me^fe^ that* raises students' motiva- 
tion toward participation in follow-up studies and improves 
availability of address information is to meet, with students 
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personally -prior to their graduation from high school. At this 
time, Jtudetits may be informed oy the^di6trict 's interest in 
periodic follow-up evaluation at/ the curriculum to "help improve 
the job preparation of later students," Students may al_sp be 
given the dites and details of s'pecific projected studies, if 
they are known. In addition, students can be asked to fill out 
a card containing their present address, the name and address 
^f their employer, and the name of one or two persons who will 
be able tO: provide information about their whereabouts in future 
years if they should change addresses. If this orientation is 
not performed before graduation, a more detailed and convincing 
orientation must be given to graduates when they are contacted 
later to participate in the follow-up study. 

The first steps in the administration of an employer 
follow-up survey are to identify former students whose employ- 
ers will be surveyed and to obtain the names and addresses of 
their employers, Not all program graduates will be employed 
and not all those employed will be working in their field of- 
training. In fact, the number of graduates employed in their 
field (and thus the number of .employers to be contacted) may be 
significantly less than the total number of program graduates 
in the target population. The size of this number will deter- 
mine whether sampling of employers is needed. 



Names and addresses of employers may be obtained through 
the following sources: > ^ 

• Follow-up questionnaires administered to program 
graduates in a concurren^t or recently completed study* 

• Phone calls or letters to program graduates or their 
pa^xents 

• Records of the School placement office 

The orientation of employers, as of students, is important 
in securing high participation rates. Employers should he 
informed of the purpose of the study and the intended uses of ,* 
the findings, convinced that their personal input is valuable^ t 
and assur.ed that their participation will not involve large 
amounts of time# The inclusion of a letter from a well-known 
member of the business community endorsing the study or a 



* An Item eliciting name and address of present employer should 
be included on these questionnaires. Graduates should also 
be asked whether their employer may be contacted for further . 
information* ^ 
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reference Co his or her name in a telephone or perfifonal contact 
may encourage employers to cooperate in the survey. 

Collecting Data 

In follow-up studies conducted by mail, the following 
items are usually included in the first mailing: a personal- 
ized letter explaining the purpose of the study and the uses to 
be made, of the findings; a questionnaire coded with the indi- 
vidual's ID number (solely for keeping track of nonrespondents) ; 
an incentive to participate, .if desired (e.g., endorsement by a 
well-known school or community figure, news clipping about the 
study, or promise of a small gift to those returning question- 
naires); and a stamped, self-addressed envelope. In the letter, 
potential participants should be assured of the confidentiality 
of the study. * 

Second and even third mailings should be made to ensure a 
high response rate. Wijth several steps in the data collection 
phase, a well-organize^F- record-keeping system is crucial to 
keep track of who has responded, who has been followed up, and 
when these steps occurred. Figure 3 provides same' tips on the 
mechanics of a mail follow-up study of former students. 

Contacting Nonrespondents 

The problem with nonrespondents is that, tjrpically, they 
represent a different sort of people than respondents. There- 
fore, vigorous efforts should be made to obtain their coopera- 
tion; follpwTup mailings and reminder phone calls with attempts 
to collect the *inforraation verbally are appropriate. An 80% 
return rate is considered necessary to reduce the possibility 
of data that are nonrepresentative of the entire population 
(Wentling, 1980). 

If this rate is not obtained through diligent follow-up of 
nonrespondents, a smaller sample of nonrespondents may be con- 
tacted and their responses compared to those obtained from the 
regular respondents. Ift^little or no difference exists between 
responses of these twof|5oups, it can be assumed, that the data 
are not biased* If signif ica^fe^diff erences are not;ed, nonre- 
spondent data should be veiTghted^according ta statistical 
weighting' procedures 'an/l then et^e^ed into the total data pool 
in an effort to correct for the exi>t^ng bias. In this case, 
aid should be solicited from an individual possessing special 
exp^ertise in this area. . v 
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FIGURE 3 ; 

/ 

Step^ in Conduccing a Data 'Colleccion by Mail 



I* Maincain former scudenci* currenc mailing addresses and ^hone 
^numbers in a card file co pennic addicions, delecions, changes, 
or sorting by program* 

2. In addition co scudenc infonnacion, include che name and address 
of parent or another iiidividual who will know scudenc 's currenc 



3. If desired) obcain from che pose office (for a fee) che new 
forwarding addresses of scudenCs who have moved. 

4. If che survey is noc anonymous, cype che scudenc 's name on che 
form. If ic ia co be anonymous, number che quescionnaire vich 
a code co represenc che respondenc so you will know who has 
recumed forms and who has noc. 

5* Use differenc colored paper for differenc programs, if daca are 
Co be sorc^d by program. 

6. Time che firsc mailing so ic will arrive mid-veek» if possible. 
Wacch ouc for holidays when mail aighc Ife 'delayed. 

7. As quescionaaires are recumed, check chem off on che card file 
and pull che card Co fild under "recums.** 

8. Some quescionnalres will come back marked "undeliverab4|(^-no 
known address." ^The pose office only keeps forvardin^ 
addresaes one year.) To update addresses for chls groups phone 
parencs, check wich inscruccors, concacc lasc knovn employer, 
or check wich younger brothers or siscers scill m school. 



9. As you are able co updace addresses, record chem on che scudenc 
cards and mail che quescionnaires. Keep chese cards separace — 
make a noCacion about when che quescionnaire was senc co che 
corrtcc address. 



10. Two weeks after the first mailing, send a second questionnaire 
and an appropriate covtr letter to all nonrtspondtnts whose 
Itttcra wire not returned by the post of f ice."^"*"^ 

11. Depending on' the number of responses obtained two weeks after 
the aecond mal).ing, you may have to send out a third mailing 
(again with a questionnaire and an appropriate cover letter) or 
use a telephone survey to contact those who have not replied. 

12* Be sure also to make second and third mailings as needed to the 
students who had corrected addresses. 



address. 
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Processing, Analyzing, and Presenting Data 

The "development of the data processing and analysis plan 
should take place at the same t'irae as the devei'opraent of the 
data collection instrument. The staff members coordinating^, 
these tasks should begin by responding to the following 
questions : 

/ 

• Should data from certain subgrouf)S be compared in the 
evaluation report (e.g., graduates versus dropouts, 
class of 1976 versus class of 1977, males versus 
females)? . ^ 

• Which items relate to the same topic area or objective 
and therefore should be analyzed together? (For 
example, data from all items relating to ^'graduates* 
present activity" could be analyzed together and a 
summary table could be developed as in Figure 4,) 

i What type of data analysis procedures^ should be used? , 
(Usually these include simple tallies, percentages, 
and averages. Examples of these appear in Figure 5. 
For cases in which more sophisticated statistical 
manipulations are needed, a. data analysis expert 
should be consulted.) 

„• How should data summaries be presented? (Methods of 
presenting findings are broadly subdivided into 
tables — numerical and s tat istical presentations — and 
figures — visual presentations. A few samples appear 
in Figure 6. ) 

Once these decisions are made, the data p^rocessing and 
analysis phase consists of the following steps': 

1. Verify all instruments for completene'ss and accuracy. 
(Errors or omissions should be corrected by data pro- 
cessing personnel, or the instrument should be returned 
to the appropriate respondent for correction.) 

2. Sort instruments into subgroups, if comparative data 
are desired. 

3. Classify and group all raw data according to topic 
area and method of statistical analysis to be used. 

4. .Record data on coding sheets for hand tabulation or 

fprmat data for keypunching or keytaping for computer 
processing, (if the survey involves .fewer than 100 
individuals, hand tallying of results is probably the^ 
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most . efficient means of summarization. Simple hand- 
^tabulation sheets may be made by cutting up a blank' 
questionnaire and pasting one or two questions op a 
page. Responses of all participants can thenJbeL^ 
recorded on the sheets.) 

Organize and record c^ta obtained from open-ended 
items (one of the most challenging steps). 

Conduct appropriate data analysis procedures manually 
or by computer;^ (Averages, percentages, or more 
sophisticated statistical tests should be calculated 
based on the predetermined data analysis plan.) 

Develop clearly labeled data summaries and displays 
(e.g., tables and figures) to present findings. 

Store or dispose of raw data from tabulated question- 
naires as desired. 
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Figure 4 








Present Accivicy of 1979 Graduates of Drafting Program 




Percent of Graduates Reporting Each Activity 


Present Activity. 


Hale 


Female 




Total 


1 —■ — 
^^EDUCATIONAL - 
Vocational School 
CoBssunityCollege 
Wear.ColXege or University 
Apprenticeship 
Other Zducation * 


42 - 
20 
5 
12 
2 
2 


.49 

18 
6 

22 
I 
2 




44 ' 
20 ' 

5 
15 

2 

2 


EMPLOYMENT 

Unemployed 

Homemake^ 

Military 
> Padd Employment ia Drafting 

Other Paid Employment 


85 

6 
0 
3 
53 
13 


33 

10 
9 
2 




85 

7 
3 

{ 7 
I49 


OTHER ACTIVIT:^ 


2 


2 




2 


TOTAL RESFOUSES 


(285) 


' (73) 




(258) 


TOTAL PERSONS REPORTING 


(244) 


(54) 




(198) 


w 

NOTE! This table is a stimmary of data from several it etas on a ijuestion- 
naire relating to "vocational graduates' present activities," 
Because many graduates were engaged in more than one activity, 
Total Respjsnses exceed Total Persons Reporting, The number of 
responses or persons is indicated in parentheses,) 
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Figure 5 

Data Analysis Procedures Useful in^FoHow-Up Studies 
Tallies \ 

How many lull-time jobs have you had since graduation from high school? 

Number of Persons Choosing Each Option (N * 50 ) 



None 

"3 ' 
5 or more 



Percentages 



3 
17 
19 
6 
4 
1 




What is the hlgliest salary you have earned on a full-time job since leaving 
t)tls vocational program? « 



Number of Persons 
Choosiiy Each Option (N » 50) 



$2.49^ or less/hr 
"$2.50 to $3.49/hr 
'$3.50 to $4.49/hr 
"$4.50 to $5.49/hr 
J5.50 to $6.49/hr 
^$6-50 or more/hr 



2 

11 
17 
11 
6 
3 



^ 2 



Percentage Choosing 
Each Option 



4 
22 
34 
22 
12 

6 



, ) 



Averages 

How well did the curriculum prepare you to use a table saw? 



Very poorly 



Satisfactorily 



Very well 





Number of Persons 




Average Rating 




Choosing Each Option (N ■ 


50) 


by Students 


I 


3 


3 




2 


0 ' 


0 




3 


1 


3 




4 


10 


40 


\ 


5 


X 9 " 


45 


5.4 


6 


13 


78 




7 


^ n 


77 




8 


3 


24 




9 


0 , 


0 

50/270 - 


5.4 



ERIC 
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Figure 6 



"^-Methods of-Fregent-lng-Data Summarlea- 
(From a Student Follow-Up Study) 



The table below presents simple tallies and ■percentages of students 
/ receiving certain startinjg salaries. 



U Tables 









Jalary Earned in First 






Fuli 


-Time Job After Graduation 








Number of 


6 






Persona 


$2,49 


or 


t 

less/hour 


8 


$2,50 


to 


$3.49/hour 


14 


$3,50 


to 


$4,49/hour 


6 


$4,50 


to 


$5,49/hour 


2 


$5,50 


to 


$6-49/hour 


2 


$6,50 


or 


more/hour 


I 



Percent of 
Students 

24 
42 
18 
'6 
6 
3 



B. In the following table, scores were averaged on four items regarding 
students* opinions of Certain aspects of the Industrial arts (wood 
shop) curriculum, and results were summarized in a single tablfe. 



Ratings Given to Wood Shop Curriculum by 
Vocational Education and Non-Vocational Education Students 



Area of Wood Shop Training 

Use of table saw 
Use oi hand saw 
Use of electric sander 
Use of lathe 

^ \ 



Ave rage 
Rating by Voc 
Ed Students 

5,4 
7.B 
6.9 

4,9 



Average Rating 
by Non-Voc 
Ed Students 

4,9 
8,1 
7,3 
4.1 
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(continued) 



Figure 6 
(continued) 



2, Figures 



A« 0£cen» Che saoe data chat are presented in, tabular (orm can also be 
presented as a figure* Below, the same data that vei^e presented in 
the table above--^the average ratings given by two groups of students 
to four aspects ^f the wood shop curriculum— »are presented as a bar • 
graphs X ^ ^ 



AVERAGE 
RATING 

BY 
STUDENTS 




ASPECT OF CURRICULUM RATED 



Line graphs are also an effective method of presenting data, parti- 
cularly .in illustrating trends over time, as in the sample below. 



r 10 



Average 
Hourly 
Vfage (in 
Dollars) 
of Voc Ed 
Graduates 
with Full- 
Time Jobs 
in 'Their 
Field of 
Training 



3 
2 



J. 




-1 \ r- 

6 12 18 

Months After Graduation 
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Individual StyiV Activities 



1. Read pages r47-178 and pages 189-208 in W6ntling, T. L. * 
Evaluating occupational education^nd training programs *. 
Boston, MA: Allyn and Bacon, 1980. 

2. The following statements describe types of item formats 
.used In data collection instruments. For each statement, 
circle T if the statement id true and F if it is false. 

X F 1. For obtaining several bits of- information, 
multiple-choice items are more economical 
in terms of time and' space than dichoto- 
mous items. 

X F 2. Dichotomous response items are used for 
cases :in which more than one option may 
apply to respondents. 

♦ 

T F 3. The- open-ended item format is one o^f the ^ 

most commonly used because it enables 
^ evaluators tp eollect large amounts of 

\ informationr Tn'^a highly efficient maijner. 



4. In a well-written ranking item, respon- 
dents are asked to rank the importance of 
long lists of clos-ely related items (e.g., 
instructional methods). 

5. Rating items are useful whe\i the evaluator 
desires ^o compare students' opinions of 
various aspects of a particular curriculum. 

6. Follow-up study instruments generally con- 
^in items of onHy one format, chosen based 
on th'e particular informational needs of 
the ^study . 
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3. Presented below is a list of tasks to be completed for an 
employer follow-up study conducted by mail. On the right*^ 
is a list of products, all of which are to be developed in 
the course of the study. Match products to the appropri- 
ate task; 'list them'in the order of their development. 
(Each task involves completion of one or more products.) 



Tasks 



Products 



1. Define the purpose 

of the study^ 

2. Prepare a work plan 

3. Select/construct 

(mail) data collection 

itistruments 

4. Select and locate the 

. part-icipants of the study 

(employers) 



L. Wri/t 



en sampling plan. 



B. List of addresses and 
phone numbers of .former 
students 

List of procedures for 
handling information 
from questionnaires 

D. Introductory letter 
explaining study' 



5. Orient participants and 
data collection staff; 
collect data 

6. Process and analyze 
data 



£• List of goals and 
objectives 

F. List of procedures for 
conducting mail survey 

G. Time/task/talent chart 



H. Summary tables and 
figures 

I. Completed staff instru- 
ment^ evaluation forms 

J« List of nances and 

addresses of ^^loyers 
of forijjer studeW^s 

k. 20-item printed 'ques- 
tionnaire 

L. List of names of for- 
mer vocational students 
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4. Summarize the data given below in response to the follow- 
ing question in both' a feble and a . Jigure , 



# i 

Question: ' 


y 


How often do you use the training you received in your 
vocational program on Jthe job?' Check one response. 


Pifequently 


Occasionally Never 


Data: 


. i 


11 students marked 
36 students marked 
3 students marked 


''Frequently" r 

"Occasionally" 

"Never" . 


Discussion Questions 





Read the following paragraph describing the place of 
follow-up studies in. the 1976 federal vocational education 
legislation: ^ . ^ 

Student and employer foll-ow-up studies are called for 
in the Vocational Education section (Title II) of the 
Education AmeriS^ents' of 1976 (P.L. 94-482). ^ Section 
112(b)('l) 6f this^legislation -requires that states 
evaluate "the extent to which program completers and ' 
leavers (i) find employment in-'occqpations related to 
their training; and (ii) ape considered by tbeir 
employers to be well-trained and prepared for 
employment." 

Then discuss the similarities and, more important! lyy the 
differences between the purposes of local follow?rfp stu-* 
dies (as described in this module) and the purposes of 
studies conducted Sy, state education aglencies in response 
to federal evaluation requirements as indicated above. 
How would the differeiq^ps affect the composition of state 
and local data collection instruments? 

Your grott^> wants to conduct a otie-year student/employer 
follc^-up evaluation of the high school electroViics curri- 
culum. Your primary^pbjectives are (1) to determine 
whether graduates are^getting jobs in the electronics - 
field and (2) to judge the value of the curriculum irt pre- 
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paring students for these jobs for the purpose of curricu- 
lum revision. You want conduct the survey by mail and 
to process data by hand sirtce no computer is available. 
One of the instruments you are considering to collect data 
from students is Jthe Virginia Department of Education's 
Student Questionnaire , reproduced on^ the next page. To 
decide whether this questionnaire is appropriate for your 
use, evaluate it according to the following steps: 

a* First analyze the instrument items and try to figure 
out the study objectives the instrument was designed 
to address. List primary objectives, subordinate 
objectives , and items relating to each subordinate 
objective. If, in your opinion, additional items are 
essential for complete coverage of th6 objective, make 
a note of them. Then determine whether a revised 
order of items is preferable. Decide whether the 
organization'of the instrument can be improved — i.e., 
can items'' be placed in a logical Sequence so that all 

i items relating to a particular objective can be 
grouped together? 

b. Then compare the instrument with the informa^tion pro- 
vided' above regarding follow-up of the electronics 
graduates, and explain whether you feel this instru- 
ment is appropriate to use in your study. If it is 
not totally suitable as is, can it be made suitable 
with minor revisions? Which ones?* 



VIRGINIA OEPARTHEKT OF EDUCATION 
VOCATIONAL EDUCATION 
FOLLOW-UP (VERS- 3) 
Student Questionnaire* 



DIRECTfuHS: Please answer each question thoughtfully. A few questions oay not apply to 
you. In such a case, oark the DNA (Does Not Apply) answer, 

USE NC). 2 PENCIL. Erase errors cotapletfly. Do not 
make .my stray laarks* • 

Do not fold, roll, staple, clip, or mutilate this fora. 
Use the LirRe self -addressed envelope to return the 
coopletcd forte. < 
No sttrpp ts "p<*dod 



Exaaples of PROPER marks 

► 00 o»o oo 



Examples of IMPROPER sarks 

0OO O(0O OO® 



1. At the present tiae, I dn 
O attending school 

O not attending school 

2. At the present tic^, I as 
O esployed as a civilian 

O In fuU-tiae nilitary service 
' O not eaployed but looking for. 
work 

O not eaployed by choice 

3. At the present tiiae, tay eaployer 
Is (pjleas? print) 

0 Dl(A ^ 

Fira or Coapany 
(If self-employed, write s elf) 



Flrt^ or Company nailing address 



My present job is- (please print) 
O DIIA 



Title 

~anB~ ttres^-ffrensy-duties t 



I an paid by the^^ 
O hour 
O 'week 
O Bonth 
O DNA 



DO NOT WRITE OR HARK IN HI IS AREA 





vsy 










CD 


O 


(D 


© ® 


© ® 


© ® 


® 


® 


® 


© 0 


© © 


© 0 


0 


® 


® 


© ® 


© © 


© © 


® 


® 


® 


© © 


® © 


© ® 






© 


© ® 


© © 


© © 


© 


© 


© 


© © 


(| © 


© © 


0 


0 


© 


0 © 


© 0 


0 ® 


® 


® 


® 


© © 


© © 


© ® 


® 


0 


® 


© © 


© © 


© ® 



The average ntnaber of 
hours I worH per week 
is (D&rken the nurabers 
that show the average 
nuffber of hours you 
work per week .) . 



© © 

© © 

© © 

© 0 

© © 

© © 

© © 

© © 

® © 

© © 



City 



State 



A. My Supervisor is (please* print) 

O DNA ^ 



7. Hy pay before 


Dollars 


Cents 


deductions 


® 


© 


© 


© 


® 


© 


for the time 




© 


© 


© 


0 


© 


period marked 


© 


© 


© 


© 


0 


0 


In Itea 6 Is 


© 


© 


© 


0 


® 


©•< 


(Darken the 


® 


® 


© 


® 


® 


© 


proper nuB- 


© 


© 


© 


© 


© 


© 


bers.) * 


© 


© 


© 


© 


© 


© 






0 


0 


0 


0 


0 




® 


© 


© 


© 


© 


© 




® 


® 


© 


® 


© 





10. 



Last naae 



First na»e 



H.l. 



♦Adapted f^oa the 1 979^80 USERS MANUAL for 
Education, Virginia Polytechnic Institute 



the VERS by D, E. Elson and S. Gcrkpn, 
and State University, Blacksburg, Vlr^ 



My present Job is related to oy 
field of vocational ecfucation 
training. 

© Yes, it is directly or closely 
related. 

© No, it is only remotely related 
or is not related. 

I was able to get this job because 
of ay vocational education training 

0 Yes 

© No 

© DNA 



RmnMBER TO COMPLETE QUESTIONS ON THE 

BACK OP Tins PAGE 

i 

Division of Vo-Tech Educatinn, CoWo^c of 
nia. 0 



(continued) 




VIRGIKIA QUESTIONHAIRE, CONTIHUED 



1 



11. At the tine that Z enrolled in a vocational education 
prograa, 

O I was able to sign up for the prograa I wanted oost. 
* O I made another choice bec8use<j^he prograa I wanted 

Dost was not offered* 
O I laade another choice because there was not a space 

for se In the prograa I wanted aost. 
O J was advised to enroll in a certain prograa because 

of ay abilities. 
O Hy parents advised ae as to which prograa I should 

take. 

O I aade ay choice because of a friend's recoenendation. 

j| O ^ assigned to the prograa. 

12. Please rate your vocational education prograa on each 
itee listed. 

1-Poor 2-Fair 3»Cood 4*Excellent 

Chances to develop technical knowledge needed 
for work 

Chances to develop aatheaatical skills needed 
for work 

Chancea to develop clerical skills needed for 
work 

Chances to develop reading skills needed for 
work 

Chances to develop cotnunicdtion skills needc 
for work 

Chances to solve probleas involving technical 

knouledg'e 

Chances to solve probleas involving work attl 
tudes 

Chances to solve probleas involving organiza- 
tion of work 

Chances to solve probleas involving huaan 
relations 



learning experiences 



A 


CP 


0 


0 




B 




0 


© 




C 


© 


0 


0 


© 


D 


© 


0 


0 


© 


E 


<I> 


0 


0 


® 


F 


O 


© 


© 


0 


G 


O 


© 


0 


© 


H 


O 


0 


0 


© 


I 


© 


0 


0 


© 


J 


O 


0 


0 


© 


K 


CD 


© 


© 


© 


L 


O 


0 


© 


© 


M 


© 


© 


0 




H 


© 




0 


© 


0 


© 


© 


© 


© 


P 


© 


0 


0 


© 


Q 


© 


© 


0 


© 



ay job 



13. My vocational training ha« aade it possible for ae to hold the 
following nuaber of full-tiae jobs: 

O 0 
O » 

O 2 

Q 3 or acre 

O Hy full-tlae jobs have not been related to ay vocational 
training. 

O I have never had a full-tlae job. 

O DNA ^ 

14. My first full-tiae job after le,nving my vocational education 
prograa 

O was related to ay training. 

O was a direct result of ay co-op experience. 

O ^ direct result of ny work/study experience. 

O was not related to ay training because no job ^as available 

O was not related to ay training because I had # good oppor- 
tunity in another field. 

O was not related to ay training because I decided that I 
didn't want to do that kind ^f work. 

O t)NA \ 

15. }iy firs t full-tiae job related to ay vocational education pro- 
graa is /was located 

O in °y own hooe coaaunity. 

O in a nearby coaaunity. ^ 

O in a distant coaaunity to which I had to move. 

O DNA 

16. I was given help in finding ay first full-tlae job after 
leaving ay vocational prograa by the person (s) aarked here. 
(Mark all that apply.) 

O vocational education teacher (s) 

O co-op teacher (s) 

O counselorCs) 

O othcjLteacher (s) 

O joKplJteraent service at the school (college) # 
O Q^her eaployment scrvicc(s) 

O triend(s) ^ 
O relative(s) 
O t)NA • 

17. May we have pcraission to shanc the information on this fora 
with your local school? ' 

O Yes 
O No 

18. May we have pcraission to contact your eap]oyor f9r informatlo 
concerning your vocational training? ' 

O , 

O No . 



Group Activity 



In small groups*, locate and interview one or two individuals 
who have been, involved in conducting a follow-up study of voca" 
tional graduates. Your interview should cover the following 
topics: 

Formulation of study goals, : v' • ' 

Development of A^ork plan (i.e., -general tasks required) 

Selection/development of the data collection instrument 

' Sample selection ^ 

Location and contact of participants and follow-up of 
nonrespond^nts 

Special problems encountered and solutions tried 
Findings of study 

Need3 for additional follow-^up studies 

Write a summarV report of the interview, including your judg- 
ments on strengths and weaknesses of the study* 

Alternative forms of' this activity include (U inviting one or 
two persons to speak to the class, interviewing them in class, 
and th6h preparing a report as described above; or (2) reading 
a summary of a vocational student or employer follow-up study 
in the professional literature and writing an evaluative- 
abstract. (Such reports are classified in the ERIC system 
under descriptors like 'Vocational follow-up,'* '*e 



patterns,'* anci "occupational suVveys.*') 



employment 
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GOAL 3: -S^ate basic guidelines fpr interpreting the results of 
V. a curriculum evaluation and for commu^nicating these 

effectively. 



Interpreting and Communicating Evaluation Results 

Once a curriculum evaluation has been conducted, conclu- 
sions must be drawn from study findings so that results may be' 
utilized for program change. Following are the activities to 
be completed by the curriculum specialist once the evaluation 
itself has been conducted: 

• Summarize background information and findings of the 
curriculum evaluation. 

• Draw conclusions from fir^^ngs and make general recom- 
mendations about future actions needed-. 

.• Summarize conclusions and recommendations and circu- >- . 
late them for comment and corrections. 



• Prepare evaluation reports for appropriate audiences 
in accordance with effective communication strategies. 

• Develop specific solutions with instructional staff 
based on study recommendations and plan for implemen- 
tation of recommendations^. 

• Monitor ' implementation of curriculum revisions. 

The last two activities listed, developing solutions and moni- 
toring their implementation, are outside the scope of this mod- 
ule, but are discus^d in the module on evaluating vocational 
education curricula and in the modules concerning curriculum 
development and curriculum change. The remainder of this mod- 
ule describes a group process approach for accomplishing the 
first three activities and presents suggestions for preparing 
and disseminating effective evaluation reports, the fourth 
activity. 
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Guidelines ^or Drawing Conpjrusions and Making -Recommendations 

Using a group apprq^h in interpreting curriculum evalua- 
tion results is advisable because it a^ows for a richness .of 
proiessional Hpe,rsE>ectives and. also- serves to promote -commitment- 
to suggested changes • The group convened to draw conclusions 
find make recommendations should' include representatives from 
the groups that might be affected by the findings oi the study: 
school administrators, curriculum advisory committee members, 
guidance and teaching personnel, and- present or former students. 
Staff members who hav^e worked on the evaluation and other per- 
sons who may contribute valuable insights based on their back- 
ground and experience should also be included. 

The members of the evaluation committee ghould be prepared 
for their work prior to the group meeting. They should receive 
a general overview of the evaluation, including the goals and . 
objectives, curriculum and population studied, data collection 
and analysis methodologies, and possible biasing factors. Pro- 
viding a copy of the work plan and data collection instruments 
is also helpful for orientation purposes. The committee should 
also be informed about how the data interpretation session will 
be conducted. ' j 

Steps to be followed in the committee meeting for drawing 
conclusions will be similar to those listed below; 



Discuss the bacjcground of the study and the specific tasks 
to be"^ccomplished . At the beginning of the session, committee 
members should be invited to ask questions and make comments 
about the study's goals and methodologies and to make general 
comments about the potential use of study findings. Guidelines 
on how study conclusions and recommendations will be derived 
should also be presented and discussed. 



Formulate a list of conclusions . The data collected in 
the study shcyuld be presented to the committee in the farm, of 
the tables and figures developed by data analysis personnel. 
All data relating to a particular objective should be grouped 
together for easier interpretation by the cocfimittee. 

* , Wentling (1980) suggests that committee members be encour- 
aged to list positive conclusions based on the results first. 
Although this may seem to be a waste of time to some, it has 
definite benefits. Listing positive conclusions will reinforce 
the constructive purposes of the evaluation and help ensure that 
the curriculum's strengths are maximized and used tc^ overcome 
its weaknesses* 
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. Then conclusions pointing out program deficiencies may be 
listed. Depending on the amount |>f time allotted for the ses- 
sion(s), the amount of data colllBcted, and the number of persons 
'present', &he group may be divided into subgroups, each of which 
will draw conclusions reigarding a particular portion of the 
data related to a particular set of objectives. 

♦Conclusions can often be drawn almost directly from 
instrument items.- Take the following instrument item: 

What did you feel was lacking in your high school 
business training? 

li many students felt that their training was lacking in ' 
the area of "office politics," a conclusion might read like 
this: / 

Many students feel that their business training did 
not prepare them properly fpr the handling of "office 
politics. " 

Data on each instrument item related to a particular . 
objective should be considered. A member of the group should 
be designated as secretary to record all proposed conclusions. 
When a list or lists of conclusions have been drafted, the 
, group shoAd evaluate their work. 

* • 

Select the best conclusions and refine their wording . The 
following criteria can be used to assess proposed criteria. 

• Is each conclusion compatible with a major objective 
of the evaluation? 

• Is there, sufficient evidence to support ^ach con- 
clusion? 

• Does each conclusion represent the best possible 
interpretation of the data? 

• Does each conclusion reflect the consensus of the 
group? 

• Is each conclusion worded in a manner that is precise 
and pe^inent to the curriculum being evaluated? 

• Does the Set of conclusions reflect all the data ^ 
collected? 
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Based on- these criteria, the group should determine which 
conclusions are most significant and most valid and should 
phrase each clearly and concisely. Although the group should 
strive for consensus on each conclusion, in some cases it may 
be appropriate *to tlist "alternate explanations of the data" 
proposed by a significant minority. 

Some of the study bindings will'be '^inconclusive." The 
group should a^oid the temptation to derive conclusions from 
every biC^ of available data, even when findings are statisti- 
cally signi^f icant. Data may sometimes' be statistically sig- 
nificant but not educationally significant, not important 
enough to require attention. On the other hand, the group may 
base tentative conclusions on "data trends" that did not reach 
statistical significance but that may suggest possible areas 
for further study. 

* 

Formulate a list of recommendations . Once the grqup has 
agreed on conclusions, it can proceed with its brainstorming, 
now with the goal of developing recommendations. At this 
point, recommendations should call for general changes in the 
curriculum and should b6 phrased using such action terms as 
"develop," "improve," "revise," "strengthen," and "add." A 
sample recommendation Ceased on the conclusion stated earlier) 
might be: ^ 

Provide training in the area of personal relations for 
business students. 

, Recommendations might include adding or discontinuing; 
units of instruction; instituting or revising prerequisites; 
totally revising curriculum objectives; modifying or expanding 
instructibnal materials, facilities, and equipment; and requir- 
ing new staff or additional stpff training. 



gelect the best recommendations and refine their wording . 
The group should then review all recommendations by asking the 
following questions: 

• Are recommendations related to major objectives of the 
evaluation? 

• Are recommendations closely linked to conclusive 
findings?. ^ 

• Do recommendations provide sufficient direction- and 
suggest meaningful actions? 
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• \ Do recommendations suggest actions that are within the 
jurisdiction of the school district and that are 
feasible in terms of resources possessed? 

Recommendations that do not warrant affirmative answers 
should be improved or deleted by the group. Then within each 
area of concern addressed by' the evaluation, recommendations 
should be ranked in order of priority from the recommendation 
that is most important to the recommendation that least merits 
immediate attention. y 

At this point., a draft summary of the group's conclusions 
and recommendations should be prepared and circulated to appro- 
priate staff for additions and corrections. The conclusions 
and recommendations should then be revised based on the input 
obtained. ' ^ 



Elements of the Evaluation Report ^ — - 

Once conclusions and recommendations have been revised or 
approved, the curriculum specialist should prepare the final 
versioji(s) of the evaluation report. Wentling (1980) notes that 
reporting is an important step in the eva^&tion process even 
though/preparing a report may seem like unnecessary 4>urden 
after^ll the effort of planning and cofcucting an evaluation. 
The evaluation report is valuable b^oause^Xt provides a Auramary 
of what has been done and can aid in promoting recommended cur- 
riculum changes. 

*" • 

The evaluation report should clearly describe the planning 
and execution of the evaluation and conveniently list the con- 
clusions and recommendations drawn from the study. \The report , 
should emphasize the significance of the results of the evalua- 
tion and call attention to the need to consider and act on pro- 
posed changes. The evaluation report, by documenting evaluation 
efforts, can provide background, guidance, and justification 
for future actionfe. 

. ' r v . 

The evaluation report can also be a useful tool in obtain- 
ing support for curriculum improvements. If properly written, 
the report can help prSVent people from^ focusing on the weak- 
nesses of the curriculum by pointing out its merits and stress- 
ing the need to build upon past achievements. For these 
reasons, reports should be prepared and distributed to those 
who may be concerned about the focus of the evaluation as well 
as those who use the curriculum that was evaluated. 

A good sample outline for afiy evaluation report appears in 
Figure 8. 



r 
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FIGURE 8 

Outline for the Curriculum Svaluation Sunantry Report * 

I. Purpose of Evaluation ^ 
k. Evaluation goals and objectives ^ . 

B» Benefits to be derived from the evaluatjh»n 
C, Audience of report ^ ^ 

II. Description of burriculun Being Evaluated 
A* Curriculum goals and objectives 

B. Description of*students 

C. Description of pro-am staff 

D» Description of instructional methods utilized 

XII. Evaluation Design and Methodology 

A. Target population and means of sai&ple selection 

B. .Rationale for evaluation ^design 

C. Schedule of events 

1 

D. Data collection methods , 

E. Data analysis metliods 

F. Limiting factors / 

/ ^ 

IV. , Evaluation Findings / 

A. Results by evaluatioh activity 

B. Results by study objective 

c 

V. Evaluation Conclusions end Recommendations 

A. Method of interpreting findings 

B. Conclusions • 

C. Recommendations i^ 

D. Suggested methods of implementing recommendations 

E. Further study needed 



Adapted from Wentli^g fl980) and Finch & Crunkilton (1979). 



^ Section I of the reJ)ort should contain a detailed discus- 
sion of the goals and objectives of the evaluation. The groups 
to be served by the study Ce,g,, voca^ion^l students and their 

* employers) shot*ldJalso be describedy^ The \specif ic audience to 
Hhora the report is addressed shouitf^be described also so that 
the reader will feel responsible for absorbing information, 
reaching conqlusions, and actively participating in implement- 
ing study-recommendations^ 

In Section ^NHHf curriculum being evaluated should be 
.defined and describe^so that the reader may better understand 
the impl-ications for actual school programs of recommendations^ 
made An the report, 'Descriptions of the socioeconomic back- 
ground and age ^oiip, male/female ratio, and vocational orien- 
tati'on of the students are al'so important, as well as infotrraa- 
tion about personnel and materials supporting the ''curriculum. 

In Section ly, wjien describing the evaluation design and 
methodalogjr, the report writer should delimit the group of stu- 
dents or emplqyers who provided Evaluation data and explain why 
this group was chosen. Sampling ^strajteg^ies should also be dis- 
cussed. The evaliiation design'-^hpuld be presented, and an 
explanation given* as to why this design was c^onsidered the most 
effective methed-of meeting the study goals. In addition, this 
"section should include a discussion x^f the f actors'TSif luen^cing-^ 
the design of the study. Economic, legal, and time restric- ^ 
tidns might be '.discussed here, A schedule of evaluation events 
should 'also be provided, ^ The Work Plan ^developed early in the 
study could serve this purpose, ^ * 

Data collection and analysis methods shou^ be^described 
so that ^the reader may determine whether conclusions are valid 
ones or whisther poorly Worded instruments, inconsistent methods 
of contacting respondents, or inaccurate statistical proc^edures 
render the findings ,untrust'worth3^. 

Limiting factors that may have ii^J-uenced* the evaluation 
results should also be outlined, Went^ling (1980) suggests dis- 
cussing 'possible shortcomings 'of the evaluation and any limit- 
ing or constraining factors that "are inherent in the results. 
It* is better^ to present these factors directly than to allow 
them to influence th^ selection or presentatidt) of results, 

• » • , - f 

The pre^ntation of evaluation^ findings in Section IV may 

be oi^anizedwi several ways. If seyeral evaluation ac^tivities 
hav€> been conducte*d at the same time (e,g,, st(jdent and employ- 
er fbllow-up studies), fiiWings could be ptesent^d for each 
activity individually, with findings grouped by study objec-' 
tives within' the discussion of each activity, ^Results could 
also be grouped- by evaluation objective* For exampl|f, findings 
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concerning student and employer assessment of job skills could 
be treated together. Finally, obsetvations oh relationships 
between findings may be presented. For eicample, you might note 
that students gave the overall vocational program an "above 
average" rating but rated most specific are^s of the curricuium 
as "average" or "below average". More subjective findiYigs may 
also be^ discussed hereby For example, you 4night\ find that ex- 
students' salaries were high, but informal contacts with'stu- • 
dents during visits to employers suggested that the students* 
job satisfaction was J.ow. 

The prd?edures Allowed in drawing conclusions and making 
recommendations should also be explained iti Sectipn V.. The 
final version Qf the conclusions and reconnnendations should be 
presented here along with suggestions for implementing study 
recomm^dations arid conducting future studies. 
/ ^ , ' 

Effective Communication Strategies 

In preparing the evaluation report, .the following steps 
should be followed': 

• Identify the reporting considerations of the particu- 
lar 5^uation X^ecisions to be made and data needed; 
^''^^•^tTdience addressed; and resources available). 

I , Determine the depth of report content, type of format, 
and Ipvel of sophistication in accordance with report- 
ing considerations. 

Use a log.ical presentation and clear,, interesting 
writing style. * ^ 

Prepare and disseminate multiple versions of the 
evaluation report as a-ppropriate. 

•» 

Reporting considerations . Initially, the evaluatQr should 
review the«jtypes of decisions to be made as a result of the 
study. ThCTe may include whether a curriculum is successful, 
whether it should' be continued or dropped, how it should be • 
revised, and whether it should be disseminated to other loca- 
tions. The information needed in a specific report' will be 
directly dependent on the decisions to be made by the report 
reader. 

Audiences\may include such groups as educational research- 
ers and senior ^administrators; board members, principals, 
teachers, and counselors; and employers, parents, and students. 
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often, the curriculum specialist will Want to address several ' 
audiences; then tie or she should consider preparing several 
reports written at different "levels of sophistication. 

Available resources are also an important reporting con- . 
sideration. A small budget may necessitate preparing a single, 
simple report that is distributed to a small number of indi- 
viduals. Time may also be a consideratidn. Impentjing dead- 
lines may require reductions in expectations regarding length 
or format. In all cases, however content; should be cut only 
as a. last resort. 

e 

Content, format, an^^^revel of sophistication . Assuming 
that resources are sufficient, it is desirable to prepJare a 
detailed report that includes all content shown in the outline 
in Figure 8. This ^/^ort should document that tKe study was 
well designed and efficiently conducted and should present data 
that clearly support the study's conclusions and recommenda- 
tions. This report, accompanied by a shorter executive summary 
of it, is particularly appropriate for commuiiicating evaluation 
results to evaluators and senior administrators. 

tfes^ight also be advisable to prepare reports and presen- 
tations specifically for other audiences. When preparing a 
report for principals, teachers, a^id' couriselbr's,' tli'e curticultrm 
specialist may be able t<J shorten or eliminate sections of thfe 
outline because these groups may be less interested in details 
of how the study was conducted and more concerned with* conc.lu- 
, sions and recommendations that directly relate to theiri. Report) 
to parents, students, and employers may be shorter ^till,- again 
with mor^ emphasis on directly relevant findings than on evalua* 
tion methodology. • ^ . . 

Possible formats for the evaluation report include a ^ 
written report; an oral presentation; an- audiovisual-- presenta- 
tion; and a printed article in a journal oi' newspaper^ The 
writ;ten report format is' best suited to the presentation of a 
large body of facts to which the reader will want to refer at a 
later date. When the results of the evaluation will directly 
and critically affect a certain educational program, the 
written report format, which put^ information "on the record," 
is the ;nost appropriate. ' 

An oral presentation can have greater immediate impact 
than a ^ritst^n report and a-llows for dialogue. However, when 
an oral presentation is used, widespread dissemination of 
information is less efficient, less material can'be covered, 
and information cannot be stored for re'fexamiriation. The oral 
.format is probably most suitable for presenting study results 
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to teachers and counselors at a meeting where discussion is 
planned and a written report is^also provided. 

The audiovisual approach, including the use of slides, 
\>verhead transparencies, moving pictures, or videobape, 
increases impact of presentation through sound, color, action, 
and immediacy* With^ the increasing availability of audiovisual 
equipment, this approach permits ready dissemination of infor- 
mation to other locations. However, letfis method is expensive, 
and does not allow dialogue with the audience. 

Articles in newspapers, magazines, professional journals, 
or newsletters offer aifother means of communicating evaluation 
results. A short article is well suited for dissemination of 
information to persons who have a g^eneral interest in the 
program but little direct impact on it. 

The level of Sophistication- refers primarily to the kinds 
of terminology and concepts used in the report ajid consequently 
to the audience for which it is appropriate. The level of 
sophistication can range from lay level, which requires little 
or *no specialized knowledge for understanding the presentation, 
to an expert level, which makes use of tecbnical terminology 
and concepts. An intermediate level is that of the board 
repott and report to teachers. The level of sophistication 
selected foi: a particular report or presentation should be 
determined by the information needs of its audience. 

Style . After determination of the appropriate content, 
format, and level, of sophistication for the evaluation report, 
the actual report writing .phase begins* A few basic principles 
of effective communication are outlined below: * 

,# Organize the repor't in a simple and logical manner 

• Use concise 'and cop;vincing language 

• Present dat^ clearly an(^ accurately ^ 

• Make your presentation interesting and attractive 

/» 

FolTowirtg an outline is crucial for good organization. 
Topic headings should be, inserted in the text or presentation 
in order ^to keep the org^izational structure clear. After the 
report is written, it should be reviewed to ensure that all 
paragraphs op sections under a certain heading are pertinent to 
t;he topic and are ordered in a* logical sequence. 
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The use^of concise language is reconnnended. Essential 
technical ^terms may be included when they will be understood* 
Unnecessarily long words and excessive jargon should be avoided 
If possible, the , report should be written in the active voice. 
Conclusions and recommendations should be precisely worded and 
action-oriented so that decision makers will be encouraged to 
receive them positively and to initiate implementation or 
recommendations.^ 

Tables and graphs can be very helpful in interpreting 
information in the 6ext and vice versa, but tables of unwieldy 
length or secondary importance should be included in ait appen- 
dix. Data collection instruments should be reproduced in 
appendices, so that readers who are interested in the metho- 
dology used in the evaluation can refer to them. 

An interesting, attractive presentation is important for 
all audiences, but gimmickjs such as novelty . typefaces or oddr 
sized' paper should be avoided. 

Dissemination^ plan . Evaluation results should almost 
invariably be distributed to certain administrative and educa- 
tional personnel and advisory group members. It may also be 
appropriate to attach a copy of the evaluation summary to an 
annual report to the State Department of Education. Individu- 
als who have been involved in designing and conducting the 
study or in providing data, to evaluators should, also have an 
opportunity to examine results. I*he general publicity value of 
curriculum evaluations is al.so high and should be realized by 
disseminating positive summary information to the media. 

^ Thus, the curriculum specialist may write a detailed ^ 
written report for fellow edifcators, give an oral or audio- 
visual presentation to the PTA, and prepare a press release for 
the local paper. In general, the wide distribution of results, 
which depends both on use of a variety of formats and the pro- 
duction of numerous copies, will increase the study visibility 
and Improve the likelihood of implementation of study recommen-* 
dations. Besides all .the decisions already mentioned, the cur- 
riculum specialist should* determine yhen varioua reports must 
be completed and how m^ny report copies shoti^LcTDe duplicated. ^ 
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* Individual Study Activities 

1. For a further discussion of methods for drawing conclusions 
from follow-up studies and of strategies for preparing 
evaluation reports, read pages 381-396* in Wentling, T, 

' Evaluating occupational education and training programs . 
Boston, MA: Allyn and Bacon, 1980. 

2. Write an agenda for a group meeting being planned to draw 
conclusions and recommendations from curriculum evaluation 

.results. List five activities for the group and the 
products that will result from each. 

3. Locate a curriculum evaluation report in ERIC or use 6ne 
with which you are familiar. Compare the report to the 
Outline for the Curriculum Evaluation Summary Report , 
Figure 8 of this module. 

Note any important information tha't is missing from the 
report or any suggested changes in organization. 

4. List eight factors to be considered in assuring effective 
communication of follow-up findings. Write a brief state- 
ment on the importance of each factor. 



Discussion Questions r * 

1. Assume you are planning the following reports of a 

^ recently completed evaluation of a carpentry curriculum, 

• A one-page flier to parents " - 

• A four-page summary to teachers 

Using the major sections of the Outline for the Curriculum 
Evaluation Summary Report > Figure 8 of this module, deter- 
mine: _ 

a. *the content elements of the outline you wi-ll include 
in each report; V 

b. the amount of space (in paragraphs) you will devote 
to each content element (assume four paragraphs per 
page) ; and ^ 

c. how you will organize the content of each report.. 
Be sure you do not go over your page limitation. 



4 
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Now suppose you want to communicate the results of the 
evaluation of the carpentry curriculum to three other 
audiences. 

• District-level administrators and evaluation staff 

• Principals and counselors 

• Graduates of the carpentry program and their employers 

In columns on a chalkboard, lfi|t for each audience above 
the following characteristics of the strategy (or strate- 
gies) you select. 

a» Format 

b. Level of sophistication 

c. Len^h . 'n 



Group Activity 

1. The data for this activity were collected as pa a six- 

month follow-up study of the graduates of a high school 
secretarial program. The number of graduates in this par- 
ticular class was 45. By phoning the few students who did 
not return the mailed questionnaire, a 100% response was 
secured. The objective of this part of the study was to 
determine how well various areas of the curriculum pre- 
pared students for their secretarial jobs. 

In this activity you will prepare data summaries for items 
on the secretarial questionnaire, and then draw conclu- 
sions and make recommendations based on the evaluation 
findings. 

a. As a group, choose an evaluation director to preside 
over the group. Together, plan'-how to group data for' 
summary presentations (e.g., presenting data from lsev- 
eral items in one figure) and decide which types of 
data analysis and presentations are appropriate for 
each set of data (tables of percentages, bar graphs of 
average ratings, etc.). Assign an individual to pre- 
pare each data presentation. The group should review 
all data presentation^ and suggest revisions as needed. 

b. As a group, choose a scribe to record conclusions and 
recommendations. The group should study the data sum- 
maries and develpp conclusions and recommendations 
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according to the steps outlined in the module (listing 
conclusions; choosing the best conclusions; listing 
recommendations; and choosing the best recommenda- 
tions). Recommendations should be directed toward 
improving ^the secretarial curriculum. 

FotTB a sub-committee composed of persons who did not 
prepare data summaries to write a summary of the con- 
clusions and recommendations developed in the meeting, 
based on the^notes taken during the session. When the 
summary is presented to the full conmittee, it should 
be revised as appropriate so that i^C represents a 
consensus of committee opinion. 
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Excerpt fron a Follow-Up Questionnaire 
Sent to Secretarial Program Graduates 
(N - 34) 



C. PAST SECkETARUL TRAlHIiiG 

17. In the following areas, now would you rate your high school 
training in term of preparing you for your secretarial jobCs)? 




7 



Job know-how, knowledge » and 












skills 














I 


Tvpewricing 


0 


7 


16 


11 


0 


2 


Shorthand 


3 


U 


17 


0 


0 


3 


Record Keeping 


0 


30 


4 


0 


0 




Filing 


0 


2 


11 


21 


0 


5 


Proofreading 


0 


24 


10 


0 


0 


6 


Spelling 


7 


20 


7 


0 


0 


7 


Punctuation 


1 


12 


13 


8 


0 


3 


Arithmetic 


I 


19 


12 


2 


0 


Use of 


tools and equipment: 












9 


Machine duplication 


0 


0 


0 


0 


5 


10 


Machine transcribing 


5 


19 


10 


0 


0 


11 


Office telephone 


0 


2 


26 


6 


0 


12 


Adding nachines 


0 


6 


25 


3 


0 


13 


Key punch 


0 


15 


11 


1 


7 



18.. If you have neld a job in Che secretarial field since leaving 

the program, how would you classify the quality of yout training 
in preparing you for the job? 

Outstanding 
Above average 
Average 
Fair 
Poor 



19. What suggestions do you have for improving .the secretarial 
program? (Cotsaents summarized below) 

7_ Obtain more modem typewriters 

4 Place more emphasis on speed in shorthand 

3~ Teach speedwriting instead of conventional shorthand 

r~ Provide more practice in machine and shorthand transcription 

l~ Give more practical experience in use of buftiness English 
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Summary 



Follow-up studies of former students and their employers 
contribute to a curriculum evaluation by providing information 
on the long-term effects of the curriculum on the knowledge and 
performance of students. This information can be used with 
information on short-term curriculum outcomes to improve the 
curriculum or judge its worth. 

Many types of decisions must be made in planning for a 
follow-up evaluation. Initial planning tasks include convening 
a meeting of selected individuals , ^choosing an evaluation 
director, and setting the scope of the study.' Once the spe- 
cific voc"5fClonal curriculum to be evaluated has been determined 
and the population of respondents has been clearly identified, 
the general goal of the follow-up study and its primary and 
subordinate objectives can be listed- 

A work plao-can then be prepared. The work plan documents 
the goal, objectives, tasks, and products of the study as well 
as the schedule, personnel, and necessary resources. The work 
plan should also specify whether personal interviews, telephone 
interviews, or mailed questionnaires will be used to collect 



When conducting a follow-up study, the curriculum special^ 
ist will need to select or develop data collection instruments; 
select the study sample; orient data collection staff; and col- 
lect, process, analyze, and display the data. 

In order for information resulting from any type o£ evalua- 
tion to be useful to decision makers; it must be effectively 
interpreted and commmunicated. A group process is valuable for 
drawing conclusions and recommendations ffom evaluation find- 
ings. Conclusions and recommendations should be presented in 
evaluation reports that discuss the purpose of the evaluation, 
the curriculum being evaluated, the evaluation design and 
methodology, and the evaluation findings. 

The content, format, and style of the, report(s) should be 
adjusted to match the information needs and interests of the - 
target audience. Widespread dissemination of evaluation results 
will improve the likelihood of implementation of study findings. 



data. 




study Activity Responses 



The answers that follow will give you an idea of the type 
of responses expected. Use thera as a study tool if you wish, 

GOAL 1 \ 
Individual Study Activities 

1. See Wentling, T. L. Evaluating occupational education and 
training programs . 

2. a. Representatives of these groups should be invited: 

• school administration 

• PTA advisory committee members 

• PTA program teachers 

• ^cnrrent PTA students 

• guidance personnel 

Having a variety of viewpoints represented at the 
planning. meeting will help ensure that the resulting 
follow-up study will collect data to address all sig- 
nificant information needs and evaluation questions. 

b. Decisions that should be made during the planning 
meeting include: 

• Who .will be the evaluation director? 

• Which PTA curriculum (what year) will be evaluated? 

• Which former students (graduates, dropouts, etc.) 
will be contacted? 

• Wi~ri bbth employers and former students be con- 
tacted? 

0 >What will be the general goal of the study? 

The evaluation directoi; should not expect primary and 
subordinate objectives to be written during the first 
meeting of the planning group. However, this group 
should review and help revise the lists of objectives, 
either individually or at another meeting. 

3. Your flow diagram should look something like the one that 
follows. Make sure you have included each of the tasks. 
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^ential 
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cipants 
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Obtain 




approval 
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for 




study 





Orient 
data col- 
lection 
staff 



00 




Pilot 
test 
instru- 
ments 



Orient 
parti- 
cipants 



\ 



Collec t 
data 



7 



Process, 




Draw con-, 




classify. 
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elusions 


If 


and 




and make 




analyz* 




recoptaen- 




data 




dations 





^Develop 
strategy 
tOs comau- 
nlcate 
findings 



Prepare 
report or 
presen- 
tation 



Follow 
up non- 
respon- 
dents 



67 



Time 



Plow Diagram of Fol low-Up Tasks 
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4. 1. T 

2. T 

3. T 

4. F ' ' 

5. T 

\ 

7. T 

8. T 

Discussion Questions 



\ 

1 



1. The response to this question will depend on your personal 
■ experiences. , ^ 

2. ^^e response to this question will depend on your personal 

experiences. ' 



GOAL 2 \ 
Individual Study Activities ' 

1. See Wentling, T. L. Evaluating occupational education and ^ 
training, programs » . . 





1. 


T 






2. 


F .J 






3. 


F 






4. 


»• 

F 








^> 


or 




5. 


T ' 














6. 


F« • . 
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I, K. ■ 
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■•9- 

63 



4. L, A, -B, 'b* * . 

5. F, D 

6. C, H 

4. Data summarized as a table 



On-the-Job Use of Vocacional Training (N-50) 









Response 
Cacegories 


Respondents 






1 


1 


Ff equenCly 


22 










Occasionally 


72 








« 


% 

Never 


6 



Data summarized as a figure 



- \ 



On-the-Job Use of Vocacional Training (.N-50) 

100 I ^ 
90 -- . 
80 -- 
70 -- 
60 



of • 

Respondents 50 

/ ^ 
^0 -- 

30 

20 " 
• 'iO 
0 



Hi 



Frequently Occasionally^ Never ^ 
Response Categories 



ERLC 



* In this example, a sample of vocational graduates was first 
identified. The former students were then contacted and 
lasked to provide the names and addresses of their employers 
and permission to contact them. 
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Discussion Question Responses ^ 

* 1* ^tate and local follow-up stud^ies are similar in that both 
are often designed to document that vocational students 
have obtained jobs in their field (student follow-up) and 
are^performing well in these jobs\( employer folJ.ow-up) . 

^ ' \ The purpose of federally mandated follow-ups, however, is 
to jui^tify past expenditures.* of federal money on stat.e 

« programs and 'to provide a rationale for continued funding. 

Hence, target populations are large and diverse (e.g., 
graduates from, all types of vocational programs in the 
state) and findings stress quantifiable outcomes (e.g., 
numbers of person^ obtaining jobs in their fields, • wages 
received, and rates of job advancement). ' > 

Persons conducting local follow-ups, on the other hand, 
often seek to assess in greater detail the work histories 
and performance of students^ of a smaller target population 
(e.g.,* all graduates of a local electronics program), for 
the purpose of making specific improvements in the local 
^ educati6nal system. Data on educational means of attain-^ 

ing job outcomes are sought (e.g., student or employer 
opinions of curricula, facilities, instruction, and 
guidance/placement services). 

4 

Items collecting information on the job history may be 
identical on $tate and local instruments. However, since 
findings of local studies wiTl result in specific remedial 
actions ot\ the curriculum level,; local follow-up instru- 
ments generally will elicit more detailed information 
^ , regarding the perceived success of a course in preparing 
students for work and regarding the actual performance of 
students on the job. 




2. a. 



Coverage of Objectives and Organization of Iteos 



Primary 
Objectives* 



Secondary 
Objectives 



Related 
Items 



Additional 
Items Needed 



Revised Order 
of Items 



T$. determine 
whether voc ed 
graduates are 
obtaining ijjbs 
n their field 
f training 



To determine 
vhether voc ed 
graduates are 
ontinuing 
their education 

To determine 
7alue of voca- 
tional programs 
In preparing 
students for 
work 



1, Are they employed? 
Full-time or part- 
time? 

2. Are they employed 
in their field of 
training? 

3. What are their job 
titles and work 
duties? 

4, What wages are they 
receiving? 

5» Where are they 
employed? 

6. How did they obtain 
their jobs? 



To determine why - 
students chose 
their vocational 
programs 

To measure graduates 
technical skills 



2,8 



5,9,10, 
13,1^ 



6,7,8 



3,4.15 



iO,l6 



In i^2, expand 
option '*not 
employed by^ 
choice" to j 
elicit reasoni 
<e,g, , nandi-l 
capped , hi6use-{ 
wife) j 

Add Item i^l9;i 

"What was 
j your field of 
i vocational 
; training?" 



Work 
history 



11 



: Add items 020i\ I 
; '%rhat kind of! 20 Education 
; education?" | 21 
■and ■/2I: '*Vhyj 
i enrolled?** ; 
] » 
I Expand options I 19 Secondary 



12A.I2F 



To obtain 
information 
for employer 
follo,w-up 
study 



J', To measure graduates* 
academic skills 

4» To measure graduates ' 
interpersonal an<| 
general occupational 
skills 

To measure quality 
of instruction 

To measure quality 
-of equipment and 
facilities 



7. 



8. 



To measure quality 
of guidan<ie and 
placement services 

To measure overall 
quality of program*' 
and inform school 
staff 



I2B,12C. 
12D 

12E,12G, 
12K,I2I 



12J,12K, 
12L 

120, 12P 



12M,12N 



12Q,1>7 



3,4,5,18 



in ?11 to I 
include "I 
enrolled in 
the prog'Tam 
for ay own 
information" 
and '1 enrolled 
CO prepare for 
5 job in the 
field." 
ICfio ^19 (above) 
would enable 
evaluation 
of success of 
soecific^ voca- 
tional prog raos 



1 1 voc ed 



12 Program 



17 



Following 
up the 
survey 



18 



*In£erred from items 
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Overall acceptability of iafitrument in meeting its 
objectives : The instrument is well-organized^ 
meets its objectives adequately, has well-written 
items in a variety of formats, and is attractive 
and easy to use. 

Suitability of instrument for own study : The 
instrtKnent is generally appropriate to the study 
target population and is suitable for use as a 
one-year mail follow-up. 

The instrument, with the addition of item 19, 
'Afield of vocational education training," ade- 
quately meets objective ^1 of the electronics 
graduates follow-up study (determining whether 
graduates are getting jobs in the electronics 
field). It also allows for collection of infor- 
mation on employers for concurrent administration 
of an employer questionnaire. 

On objective #2 (judging the value and revising 
the curriculum), the instrument is weaker^ Since 
it was designed to survey graduates of all types 
of programs, the items related to curriculum, 
while numerous, serve only for general assessment 
of a particular curriculum. More detailed 
assessment of the electronics curriculum could be 
achieved, however, by revising item #12 of the 
instrument. For example, item 12A, ^'chances to 
develop technical knowledge needed for work," 
could be expanded to read "chances to^develop . 
knowledge and skills i^n (a) using soldering 
tools; (b) operating drill press, (c) using 
oscilloscope , (d) installing > testing , and con-^ 
structing PC boards, (e) using transistor/tube/ 
instrument manuals," -etc. 

Also, the instrument could be easily processed by 
hand. To facilitate data processing, the format 
of items 8 and 9 could be changed to a multiple-, 
choice format with each option containing a range 
of salaries or working hours (e.g., $3-4 per hour 
or 35-40 hours per week); 



«r 
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GOAL 3 



[ndividual Study Activities 

See Want ling, T. L. Evaluating occupati^al education and 
training programs . 

• Activity : Discuss the background of the study and 
the specific tasks to be accomplished 

Product : Well-oriented evaluation committee members 
prepared to work 

• Activity ; Formulate a list of conclusions 

Product : A draft list of positive conclusions and 
conclusions pointing out program defi- 
ciencies 



• Activity : 



Product : 



Activity : 
Product : 



Select the best conclusions and refine 
their wording 

A refined list of clear, concise, sig- 
nificant, and valid conclusions that meet 
quality criteria 

Formulate ^a list of recommendations 

A draft list of recommendations phrased in 
action terms 



• Activity : Select the best recommendations and refine 
their wording 

Product : A refined list of high quality recommen- 
dations ranked in order of priority 

3. Your response will depend on the report you locate. 

Factors to be considered in preparing the evaluation 
report include the following: 



Decisions to be made by the reader and data needed 

Audience addressed 

Resources available - ' 

Content of report 

i^ormat 

Level of sophistication 
Style 

Dissemination plans 
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The logical end of a curriculum evaluation is the adop- 
tion of recommendations by appropriate educational per- 
sonnel. ThuSj it is important that the evaluation report 
concentrate on the decisions that need to be made as a 
result of the study and, specifically, on those decisions 
that will be made by the group^receiving the report. In 
other words, the writer should know the audience he or she 
is addressing and the kind of information needed in order 
for this group to take responsible actions in improving 
the curriculum. If constraints in time, money, and 
staff — obvious important considerations in planning the 
elaborateness of the report — are recognized early, care 
can be taken to simplify format, level of sophistication, 
and dissemination plans without reducing content. 

Report content should follow a basic outline for all audi- 
ences, since a certain logical flow of information is 
essential to all audiences. Content, however, can be 
treated to varying degrees of depth depending on the audi- 
ence. Format should be decided on based on the audience 
to be addressed and the purpose of the report. Choosing 
an appropriate format (e.g., written, oral, or audiovisual) 
- is important in order to maximize the audience's interest 
in the content. 

Level of sophistication of terminology may vary from the 
expert to the lay level. Use of overly technical language 
in a report to a particular group may result in lack of 
interest in report findings and subsequent failure to 
implement recommendations. { 

Good style is not optional in any report. All reports 
should be well-organized, interesting, attractive, con- 
cise, convincing, and clelr. Dissemination plans are 
important to ensure that various groups obtain needed 
information that will enable them to take certain actions 
in response to evaluation findings* 

Discussion Question Responses 

1. Parents will most want to know whether the curriculum was 
effective and it? benefits to their children. In a one- 
page flier you should devote one paragraph to the purpose 
of the evaluation, description of the curriculum, and 
evaluation design and methodology; two paragraphs to the 
evaluation findings; and one paragraph to conclusions and 
recommendations. These topics should be discussed in this 
order. 
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Teachers will most want to know about recommended changes 
in the curriculum and the findings that support the recom- 
mendation?. One paragraph should be devoted to the purpose 
of the ''evaluation. A lengthy description of the curriculum 
is not necessary; one paragraph will do. Two paragraphs 
should be used to describe the evaluation' design and 
methodology. Yhis would complete the first page of the 
report. The remaining three pages should be assigned to 
findings, conclusions, and recommendations. It would be 
advisable' to arrange the text so that supporting data 
(findings) were discussed after each recommendation. 
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Plan for Preparing and Disseminating Infonaation 
on an Evaluation of a Carpentry Currlcuiua 



^ Target 
Audier.ce 


Format 


Level of \ 
Sophistication 


Length 


■ 

I. District level 


Written report 


Expert 


15-20 pages 


adciinis t rat ors 
and evaluation 


and 






staff 


Executive Sunaary , 




3 pages 


2, Principals and 
counselors * 


0\al presentation 
accompanied by 
Written report / 


Board 

If 


I hour 
(discussion 
to follow) 

5 pages 


3. Graduates of 


Newspaper article 


Lay 


5 paragraphs 


the carpentry 
prograa and 
cneir employers 


i 
1 






« 














t 




/ 






* 




* 
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Self-Check 



GOAL 1 



1. State the purpose for conducting a follow-up study as part 
of a curriculum evaluation* 

2. List the three steps in the process of establishing the 
-^t^^obj^ectives of a particular follow-up evaluation. 

3* List the six components of an evaluation study work plan. 

4, Give two reasons why the personal interview technique may 
be valuable in surveying employers of vocational graduates. 



1. In selecting items to be included in a data collection 
instrument, the curriculum specialist should determine 
whether they '»pertaiTi to the . 

2. List at least five criteria for evaluating existing data 
collection instruments for possible use in*a student 
follow-up study. 

3. List at least four guidelines for developing a mail ques- 
tionnaire that will help to ensure a high response rate 
from participants. 

4. In conducting a mail survey, which of the following data 
collection schedules would be most appropriate? 

a. A single mailing of the questionnaire 

b. A single mailing, then phone calls to Bonrespondeiils 

c. Two mailings 

d. Two mailings^ then phone calls to nonrespotidents 

e. Three or more mailings 

5. List four decisions that must be made in developing a data 
^ analysis plan. 



GOAL 2 
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GOAL 3 



1. 



2. 



3. 



4. 



5. 



Lisjt five steps involved in the evaluation committee 
appfeach to interpreting results of follow-up studies. 

List at least five criteria to bemused in developing a 
final set of conclusions based on follow-up data. 

Before sections discussing the research design and metho- 
dology in an evaluation report, sections should appear 
describing the and the 



If report preparation funds are limited, the report format 
that probably should be ruled out first is the: 

\ 

a. written report 

b. oral presentation ' 

c . audiovisual presentation 

d. journafl article 

Read the following examples of inappropriate behaviors by 
report writers and indicate the kind of error made in each 
situation (e.*g., poor choice of audience). 



Behavior 



Kind of Etror 



a. Developing a newspaper article 
to inform school statisticians 
of study findings 

b. In a brochure to parents, 
**refuting the hypothesis" of 
the' study byuse of "infer-^ 
ential statistics*' 




c. Omitting information regarding 
problems of, data collection from 

•a report to evaluation experts 

d. Presenting information to parents 
about various sampling options 
considered in the study 

A 

e. Placing all tables artd^ figures 
in the appendix of a written 
report 



-89- 



73 



Self-Check Responses 



The answers that follow ^ill give you an idea of the type 
of rersponse expected. Use thera as a study tool if you wish. 



GOAL 1 
1. 



3. 



4. 



To judge the value of the curriculum in producing long- 
erm outcomes and to alter it as needed 

Convene a planning meeting and select an evaluation 
director 

Define the 3Cope of the study 

Write the general goal of the follow-up and list pri- 
mary and subordinate objectives 

Study goals and objectives 

Tasks and subtasks 

Products 

Completion dates 
Individuals responsible 
Resources needed 

In-depth information regarding employers' perceptions 
of graduates' job performance and recommendations for 
curriculum revision may be obtained. 



• Personal contact with employers may also serve to pro- 
mote job placement of additional vocational graduates. 



i 



GOAL 2 

1. Pre-established goals and objectives of the study 

2. Any five of the following criteria are acceptable 



I 

• Is it readily available and inexpensive? 

• Is it directed at the appropriate target population? 
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Is it directed at the appropriate goals and objectives? 

Is it designed for the appropriate time frame and 
method of data collection? * 

/ 

Is item format appropriate and style clear and concise? 



listered in aVe 



Can it be administered in. a Treasonable amount'of time? 
Can data be processed easily? 
Is the instrument reliable? 



Is the instrument valid? 

' \ . 

four of the following resrponses are acceptable: 

Word items clearly to avoid confusion. 

Provide clear instructions on how to respond to items. 

Vary item format to enhance interest. 

Avoid asking for unnecessary information or informa- 
tion that could easily be obtained in another way. 
(Keep questionnaire shorti) 

Avoid asking confidential or sensitive questions. 

Use open-ended items sparingly. Where possible, use 
mul^feipLe-choice items or other item* formats where 
options are provided and respondents must simply check 
the preferred one. 

Design the questionnaire, to have an oj^jierly layout and 
attractive appearance (e.g., use colo^d paper) .'^ 

V 



WheCFier data should be processed by subpopulations 
(male versu's^ female, graduates versus dropouts , etc. ) " 

Which items are related in content and should be 
analyzed , together 

What kinds of d'ata analys^es should be performed on data 

\ * . 

How data summaries should be. presented 
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GOAL 3 . 

1. • Discuss bat:kgr9und of the study and the task to be ^ 

accomplished by the conunittee. 

« 

• Formulate a list of all possible conclusions. 

• Select the best conclusions and refine their wordVng. 

• Formulate a list of all possible req^jwnendations*! 

• Select the best recommendations and refine their 
wording. 

2. Any five of the following criteria are acceptable:^ 

• Is each conclusion compatible with a major study 
objective? 

• Is there sufficient evidence to support each con- 
clusion? 

• Doe^ each conclusion represent the best pbssible 
interpretation of the data? 

• Does each conclusion reflect the consensus of the 
group? ^ 

• Is the wording of each conclusion precise and 
curriculum-oriented? ^ 

• Does the set of conclusions comprehensively interpret 
all data collected? 

* * 

3. • • . purpose of the evaluation and the curriculum being 
^ evaluated 

' h 

4. c 

5. a. inappropriat,e report format 

b. wrong level of sophistication (too technical) 
d. ^inadequate content ^ 

d, wrong audience 

e. poor style 
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VECS Module Titles 



Module 1: Vocational Educa|Cors and Curriculum Management 



Modifle 


2: 


The Scope of Vocational Education 


Module 


3: 


f 

Organization of Vocational Educ^ation 


Hodule 


4: 


Legislative Mandates for Vocational Education 


Module 


5: 


Priorities in Vocational Education 


Module 


6: 


"Vocational Education for Students with Special Needs 


Module 


7: 


Vocational Needs Assessment and Curriculum Devel- 
opment 


Module 


8: 


Conducting Task Analyses and Developing Instruc- ^ 
tionart Objectives ^ _^ 


Module 


9: 


Selecting Instructional Strategies and Assessing 
' Student Achievement 


Module 


10: 


Relating Learning Differences and Instructional 
Methods ^ * 


Module 


11: 


Selecting and Preparing Pfisrructional Materials 


Module 


12: 


Evaluating Vocational Education Curricula 


Module 


13: 


Conducting Follow-Up Studies and Communicating 
Evaluation Results 


Module 


14: 


Managing Vocational Education Programs ^ 


Module 


15: 


Preparing for Curriculum Change 


Module 


16: 


Staff Development • 
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